From noreply at bro.org  Mon Sep  1 00:00:21 2014
From: noreply at bro.org (Merge Tracker)
Date: Mon, 1 Sep 2014 00:00:21 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409010700.s8170LSA014940@bro-ids.icir.org>


Open Merge Requests
===================

ID            Component    Reporter    Assignee    Updated     For Version    Priority    Summary
------------  -----------  ----------  ----------  ----------  -------------  ----------  ----------------------------------------
BIT-1231 [1]  Bro          hui         hui         2014-08-29  -              Normal      DNP3 Analyzer Supports for DNP3-over-UDP


[1]  BIT-1231  https://bro-tracker.atlassian.net/browse/BIT-1231


From noreply at bro.org  Tue Sep  2 00:00:21 2014
From: noreply at bro.org (Merge Tracker)
Date: Tue, 2 Sep 2014 00:00:21 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409020700.s8270LxL015918@bro-ids.icir.org>


Open Merge Requests
===================

ID            Component    Reporter    Assignee    Updated     For Version    Priority    Summary
------------  -----------  ----------  ----------  ----------  -------------  ----------  ----------------------------------------
BIT-1231 [1]  Bro          hui         hui         2014-08-29  -              Normal      DNP3 Analyzer Supports for DNP3-over-UDP


[1]  BIT-1231  https://bro-tracker.atlassian.net/browse/BIT-1231


From jira at bro-tracker.atlassian.net  Tue Sep  2 04:57:07 2014
From: jira at bro-tracker.atlassian.net (Brian O'Berry (JIRA))
Date: Tue, 2 Sep 2014 06:57:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1238) High false-positive for
 application/x-tar signature
In-Reply-To: <JIRA.15800.1409658967000@Atlassian.JIRA>
References: <JIRA.15800.1409658967000@Atlassian.JIRA>
	<JIRA.15800.1409658967822@bro-tracker.atlassian.net>
Message-ID: <JIRA.15800.1409658967000.1.1409659027684@Atlassian.JIRA>

Brian O'Berry created BIT-1238:
----------------------------------

             Summary: High false-positive for application/x-tar signature
                 Key: BIT-1238
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1238
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: 2.3
            Reporter: Brian O'Berry


The following signature in base/frameworks/files/magic/general.sig frequently triggers on text files in our environment, and includes a strength value higher than GNU and POSIX tar signatures in libmagic.sig.
{code}
signature file-tar {
    file-magic /([[:print:]\x00]){100}(([[:digit:]\x00\x20]){8}){3}/
    file-mime "application/x-tar", 150
}
{code}



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From seth at icir.org  Tue Sep  2 06:53:30 2014
From: seth at icir.org (Seth Hall)
Date: Tue, 2 Sep 2014 09:53:30 -0400
Subject: [Bro-Dev] Adding a LOCAL option to the Direction type?
In-Reply-To: <CANUJ_Fn7EvmXSSV+5_RGRpTy8JpUHh+Y86JY-DjSX2TrhZec3w@mail.gmail.com>
References: <CANUJ_Fn7EvmXSSV+5_RGRpTy8JpUHh+Y86JY-DjSX2TrhZec3w@mail.gmail.com>
Message-ID: <349E681A-27B2-40D2-A5F1-C66C6452D2C7@icir.org>


On Aug 28, 2014, at 7:16 PM, Vlad Grigorescu <vlad at grigorescu.org> wrote:

> Does it make sense to add LOCAL == local orig, local resp? Similarly, do we want to add EXTERNAL == remote orig, remote resp?

Yes, I've been meaning to add this for years.  Please do so. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140902/36a5c1ba/attachment.bin 

From noreply at bro.org  Wed Sep  3 00:00:38 2014
From: noreply at bro.org (Merge Tracker)
Date: Wed, 3 Sep 2014 00:00:38 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409030700.s8370cwl011784@bro-ids.icir.org>


Open Merge Requests
===================

ID            Component    Reporter    Assignee    Updated     For Version    Priority    Summary
------------  -----------  ----------  ----------  ----------  -------------  ----------  ----------------------------------------
BIT-1231 [1]  Bro          hui         hui         2014-08-29  -              Normal      DNP3 Analyzer Supports for DNP3-over-UDP


[1]  BIT-1231  https://bro-tracker.atlassian.net/browse/BIT-1231


From noreply at bro.org  Thu Sep  4 00:00:19 2014
From: noreply at bro.org (Merge Tracker)
Date: Thu, 4 Sep 2014 00:00:19 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409040700.s8470J0F019280@bro-ids.icir.org>


Open Merge Requests
===================

ID            Component    Reporter    Assignee    Updated     For Version    Priority    Summary
------------  -----------  ----------  ----------  ----------  -------------  ----------  ----------------------------------------
BIT-1231 [1]  Bro          hui         hui         2014-08-29  -              Normal      DNP3 Analyzer Supports for DNP3-over-UDP


[1]  BIT-1231  https://bro-tracker.atlassian.net/browse/BIT-1231


From jira at bro-tracker.atlassian.net  Thu Sep  4 09:36:07 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Thu, 4 Sep 2014 11:36:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1231) DNP3 Analyzer Supports for DNP3-over-UDP
In-Reply-To: <JIRA.15400.1408049217000@Atlassian.JIRA>
References: <JIRA.15400.1408049217000@Atlassian.JIRA>
	<JIRA.15400.1408049217421@bro-tracker.atlassian.net>
Message-ID: <JIRA.15400.1408049217000.3.1409848567737@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robin Sommer reassigned BIT-1231:
---------------------------------

    Assignee: Robin Sommer  (was: hui)

> DNP3 Analyzer Supports for DNP3-over-UDP
> ----------------------------------------
>
>                 Key: BIT-1231
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1231
>             Project: Bro Issue Tracker
>          Issue Type: Improvement
>          Components: Bro
>    Affects Versions: 2.3
>            Reporter: hui
>            Assignee: Robin Sommer
>              Labels: DNP3, analyzer
>
> Two major changes are made for the DNP3 analyzer
> 1. Make the analyzer support both the DNP3-over-UDP and the DNP3-over-TCP.
>     The changes are made in DNP3.cc, DNP3.h and dpd.sig
> 2. Fix a bug in the binpac codes of the DNP3 analyzer
>    The changes are made in dnp3-protocol.pac. The changes results in different baseline results of testing/btest/Baseline/scripts.base.protocols.dnp3.dnp3_link_only
>  



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Thu Sep  4 12:38:07 2014
From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA))
Date: Thu, 4 Sep 2014 14:38:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1212) Segfault in X509 file analyzer
In-Reply-To: <JIRA.14800.1404134746000@Atlassian.JIRA>
References: <JIRA.14800.1404134746000@Atlassian.JIRA>
	<JIRA.14800.1404134746711@bro-tracker.atlassian.net>
Message-ID: <JIRA.14800.1404134746000.7.1409859487826@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Johanna Amann updated BIT-1212:
-------------------------------
    Status: Merge Request  (was: Open)

> Segfault in X509 file analyzer
> ------------------------------
>
>                 Key: BIT-1212
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1212
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: 2.3
>            Reporter: Marek Balint
>            Assignee: Johanna Amann
>
> Bro segfaults in src/file_analysis/x509/functions.bif:256, due to base->certs being NULL.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Thu Sep  4 12:38:07 2014
From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA))
Date: Thu, 4 Sep 2014 14:38:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1212) Segfault in X509 file analyzer
In-Reply-To: <JIRA.14800.1404134746000@Atlassian.JIRA>
References: <JIRA.14800.1404134746000@Atlassian.JIRA>
	<JIRA.14800.1404134746711@bro-tracker.atlassian.net>
Message-ID: <JIRA.14800.1404134746000.5.1409859487732@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17900#comment-17900 ] 

Johanna Amann commented on BIT-1212:
------------------------------------

Bug is fixed in topic/johanna/ticket-1212, including a few other fixes that could result in wrong validation results.

If we do a .1 release, we should also include this. It should not be exploitable (in the worst case it is a 0-pointer dereference), but it makes the whole functionality completely useless.

> Segfault in X509 file analyzer
> ------------------------------
>
>                 Key: BIT-1212
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1212
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: 2.3
>            Reporter: Marek Balint
>            Assignee: Johanna Amann
>
> Bro segfaults in src/file_analysis/x509/functions.bif:256, due to base->certs being NULL.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Thu Sep  4 12:41:07 2014
From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA))
Date: Thu, 4 Sep 2014 14:41:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1212) Segfault in X509 file analyzer
In-Reply-To: <JIRA.14800.1404134746000@Atlassian.JIRA>
References: <JIRA.14800.1404134746000@Atlassian.JIRA>
	<JIRA.14800.1404134746711@bro-tracker.atlassian.net>
Message-ID: <JIRA.14800.1404134746000.10.1409859667781@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Johanna Amann updated BIT-1212:
-------------------------------
    Assignee: Robin Sommer  (was: Johanna Amann)

> Segfault in X509 file analyzer
> ------------------------------
>
>                 Key: BIT-1212
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1212
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: 2.3
>            Reporter: Marek Balint
>            Assignee: Robin Sommer
>
> Bro segfaults in src/file_analysis/x509/functions.bif:256, due to base->certs being NULL.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Thu Sep  4 16:56:07 2014
From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA))
Date: Thu, 4 Sep 2014 18:56:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1239) Crash because StringVal ref_cnt greater
 than INT_MAX
In-Reply-To: <JIRA.15801.1409874952000@Atlassian.JIRA>
References: <JIRA.15801.1409874952000@Atlassian.JIRA>
	<JIRA.15801.1409874952733@bro-tracker.atlassian.net>
Message-ID: <JIRA.15801.1409874952000.11.1409874967645@Atlassian.JIRA>

Johanna Amann created BIT-1239:
----------------------------------

             Summary: Crash because StringVal ref_cnt greater than INT_MAX
                 Key: BIT-1239
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1239
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: 2.3, git/master
            Reporter: Johanna Amann
             Fix For: 2.4


Several of the workers of our cluster recently crashed because of the compare in line 212 of Obj.h (function Ref, where o->ref_cnt is compared to INT_MAX).

Closer examination of the stack traces of a few systems reveals that it was the StringVal base_type which was ref'd more than INT_MAX times.

In the last few times, a user in our network performed tests generating a massive amount of connections, including test with just syn-packets. It is therefore probably a good guess that somewhere in the connection handling, a ref on a base_type is called without an corresponding unref.

Relevant part of a backtrace:

{code}
#0  0x000000080194cfcc in kill () from /lib/libc.so.7
#1  0x000000080194bdcb in abort () from /lib/libc.so.7
#2  0x00000000004ca550 in Reporter::InternalError (this=)
at /home/robin/bro/master/src/Reporter.cc:137
#3  0x00000000004d1d79 in bad_ref (type=)
at /home/robin/bro/master/src/Obj.cc:253
#4  0x00000000005273cd in base_type (tag=) at Obj.h:208
#5  0x0000000000538d8b in StringVal (this=0x87abca240, length=66,
s=0x85b5e3290 "[...]") at Val.h:369
{code}

stderr.log:
{code}
1406688636.803846 processing suspended
1406688636.803849 processing continued
1406688642.801786 Failed to open GeoIP Cityv6 database: /usr/local/share/GeoIP/GeoIPCityv6.dat
1406688642.801786 Failed to open GeoIPv6 Country database: /usr/local/share/GeoIP/GeoIPv6.dat
1409680506.073977 internal error: bad reference count [1]
/xa/bro/share/broctl/scripts/run-bro: line 85: 98029 Abort trap: 6           (core dumped) nohup $mybro "$@"
{code}



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Thu Sep  4 21:17:07 2014
From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA))
Date: Thu, 4 Sep 2014 23:17:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1212) Segfault in X509 file analyzer
In-Reply-To: <JIRA.14800.1404134746000@Atlassian.JIRA>
References: <JIRA.14800.1404134746000@Atlassian.JIRA>
	<JIRA.14800.1404134746711@bro-tracker.atlassian.net>
Message-ID: <JIRA.14800.1404134746000.13.1409890627719@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Johanna Amann updated BIT-1212:
-------------------------------
    Fix Version/s: git/master
           Status: Closed  (was: Merge Request)

> Segfault in X509 file analyzer
> ------------------------------
>
>                 Key: BIT-1212
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1212
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: 2.3
>            Reporter: Marek Balint
>            Assignee: Robin Sommer
>             Fix For: git/master
>
>
> Bro segfaults in src/file_analysis/x509/functions.bif:256, due to base->certs being NULL.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From noreply at bro.org  Fri Sep  5 00:00:27 2014
From: noreply at bro.org (Merge Tracker)
Date: Fri, 5 Sep 2014 00:00:27 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409050700.s8570RRW025668@bro-ids.icir.org>


Open Merge Requests
===================

ID            Component    Reporter    Assignee      Updated     For Version    Priority    Summary
------------  -----------  ----------  ------------  ----------  -------------  ----------  ----------------------------------------
BIT-1231 [1]  Bro          hui         Robin Sommer  2014-09-04  -              Normal      DNP3 Analyzer Supports for DNP3-over-UDP


Open Fastpath Commits
======================

Commit       Component    Author         Date        Summary
-----------  -----------  -------------  ----------  --------------------
f01e862 [2]  bro          Johanna Amann  2014-09-04  fix more http links.


[1]  BIT-1231  https://bro-tracker.atlassian.net/browse/BIT-1231
[2]  f01e862   https://github.com/bro/bro/commit/f01e8629fcb6acec5001bc2f55a396e174228a9a


From jira at bro-tracker.atlassian.net  Fri Sep  5 11:32:08 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Fri, 5 Sep 2014 13:32:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1239) Crash because StringVal ref_cnt greater
 than INT_MAX
In-Reply-To: <JIRA.15801.1409874952000@Atlassian.JIRA>
References: <JIRA.15801.1409874952000@Atlassian.JIRA>
	<JIRA.15801.1409874952733@bro-tracker.atlassian.net>
Message-ID: <JIRA.15801.1409874952000.16.1409941928344@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17901#comment-17901 ] 

Jon Siwek commented on BIT-1239:
--------------------------------

Does this affect git/master or just 2.3 ?

Looking quickly at base_type usages, I didn't see anything suspicious.  But I did fix one problem in commit bfaa082aeeb1b5e4ffc51a912008ac9cc6409d72, which is in master, but not 2.3.

> Crash because StringVal ref_cnt greater than INT_MAX
> ----------------------------------------------------
>
>                 Key: BIT-1239
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1239
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>            Reporter: Johanna Amann
>             Fix For: 2.4
>
>
> Several of the workers of our cluster recently crashed because of the compare in line 212 of Obj.h (function Ref, where o->ref_cnt is compared to INT_MAX).
> Closer examination of the stack traces of a few systems reveals that it was the StringVal base_type which was ref'd more than INT_MAX times.
> In the last few times, a user in our network performed tests generating a massive amount of connections, including test with just syn-packets. It is therefore probably a good guess that somewhere in the connection handling, a ref on a base_type is called without an corresponding unref.
> Relevant part of a backtrace:
> {code}
> #0  0x000000080194cfcc in kill () from /lib/libc.so.7
> #1  0x000000080194bdcb in abort () from /lib/libc.so.7
> #2  0x00000000004ca550 in Reporter::InternalError (this=)
> at /home/robin/bro/master/src/Reporter.cc:137
> #3  0x00000000004d1d79 in bad_ref (type=)
> at /home/robin/bro/master/src/Obj.cc:253
> #4  0x00000000005273cd in base_type (tag=) at Obj.h:208
> #5  0x0000000000538d8b in StringVal (this=0x87abca240, length=66,
> s=0x85b5e3290 "[...]") at Val.h:369
> {code}
> stderr.log:
> {code}
> 1406688636.803846 processing suspended
> 1406688636.803849 processing continued
> 1406688642.801786 Failed to open GeoIP Cityv6 database: /usr/local/share/GeoIP/GeoIPCityv6.dat
> 1406688642.801786 Failed to open GeoIPv6 Country database: /usr/local/share/GeoIP/GeoIPv6.dat
> 1409680506.073977 internal error: bad reference count [1]
> /xa/bro/share/broctl/scripts/run-bro: line 85: 98029 Abort trap: 6           (core dumped) nohup $mybro "$@"
> {code}



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From robin at icir.org  Fri Sep  5 15:04:23 2014
From: robin at icir.org (Robin Sommer)
Date: Fri, 5 Sep 2014 15:04:23 -0700
Subject: [Bro-Dev] [JIRA] (BIT-1239) Crash because StringVal
	ref_cnt	greater than INT_MAX
In-Reply-To: <JIRA.15801.1409874952000.16.1409941928344@Atlassian.JIRA>
References: <JIRA.15801.1409874952000@Atlassian.JIRA>
	<JIRA.15801.1409874952733@bro-tracker.atlassian.net>
	<JIRA.15801.1409874952000.16.1409941928344@Atlassian.JIRA>
Message-ID: <20140905220423.GQ79275@icir.org>



On Fri, Sep 05, 2014 at 13:32 -0500, you wrote:

> Does this affect git/master or just 2.3 ?

2.3, so yeah that's a possible explanation, though hard to say.



From jira at bro-tracker.atlassian.net  Fri Sep  5 15:05:07 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Fri, 5 Sep 2014 17:05:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1239) Crash because StringVal ref_cnt greater
 than INT_MAX
In-Reply-To: <JIRA.15801.1409874952000@Atlassian.JIRA>
References: <JIRA.15801.1409874952000@Atlassian.JIRA>
	<JIRA.15801.1409874952733@bro-tracker.atlassian.net>
Message-ID: <JIRA.15801.1409874952000.18.1409954707712@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17902#comment-17902 ] 

Robin Sommer commented on BIT-1239:
-----------------------------------





2.3, so yeah that's a possible explanation, though hard to say.




> Crash because StringVal ref_cnt greater than INT_MAX
> ----------------------------------------------------
>
>                 Key: BIT-1239
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1239
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>            Reporter: Johanna Amann
>             Fix For: 2.4
>
>
> Several of the workers of our cluster recently crashed because of the compare in line 212 of Obj.h (function Ref, where o->ref_cnt is compared to INT_MAX).
> Closer examination of the stack traces of a few systems reveals that it was the StringVal base_type which was ref'd more than INT_MAX times.
> In the last few times, a user in our network performed tests generating a massive amount of connections, including test with just syn-packets. It is therefore probably a good guess that somewhere in the connection handling, a ref on a base_type is called without an corresponding unref.
> Relevant part of a backtrace:
> {code}
> #0  0x000000080194cfcc in kill () from /lib/libc.so.7
> #1  0x000000080194bdcb in abort () from /lib/libc.so.7
> #2  0x00000000004ca550 in Reporter::InternalError (this=)
> at /home/robin/bro/master/src/Reporter.cc:137
> #3  0x00000000004d1d79 in bad_ref (type=)
> at /home/robin/bro/master/src/Obj.cc:253
> #4  0x00000000005273cd in base_type (tag=) at Obj.h:208
> #5  0x0000000000538d8b in StringVal (this=0x87abca240, length=66,
> s=0x85b5e3290 "[...]") at Val.h:369
> {code}
> stderr.log:
> {code}
> 1406688636.803846 processing suspended
> 1406688636.803849 processing continued
> 1406688642.801786 Failed to open GeoIP Cityv6 database: /usr/local/share/GeoIP/GeoIPCityv6.dat
> 1406688642.801786 Failed to open GeoIPv6 Country database: /usr/local/share/GeoIP/GeoIPv6.dat
> 1409680506.073977 internal error: bad reference count [1]
> /xa/bro/share/broctl/scripts/run-bro: line 85: 98029 Abort trap: 6           (core dumped) nohup $mybro "$@"
> {code}



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From noreply at bro.org  Sat Sep  6 00:00:36 2014
From: noreply at bro.org (Merge Tracker)
Date: Sat, 6 Sep 2014 00:00:36 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409060700.s8670aPn022546@bro-ids.icir.org>


Open Merge Requests
===================

ID            Component    Reporter    Assignee      Updated     For Version    Priority    Summary
------------  -----------  ----------  ------------  ----------  -------------  ----------  ----------------------------------------
BIT-1231 [1]  Bro          hui         Robin Sommer  2014-09-04  -              Normal      DNP3 Analyzer Supports for DNP3-over-UDP


Open Fastpath Commits
======================

Commit       Component    Author         Date        Summary
-----------  -----------  -------------  ----------  --------------------
f01e862 [2]  bro          Johanna Amann  2014-09-04  fix more http links.


[1]  BIT-1231  https://bro-tracker.atlassian.net/browse/BIT-1231
[2]  f01e862   https://github.com/bro/bro/commit/f01e8629fcb6acec5001bc2f55a396e174228a9a


From jira at bro-tracker.atlassian.net  Sat Sep  6 13:46:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Sat, 6 Sep 2014 15:46:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1240) TCP gaps inserted in wrong place
In-Reply-To: <JIRA.15802.1410036351000@Atlassian.JIRA>
References: <JIRA.15802.1410036351000@Atlassian.JIRA>
	<JIRA.15802.1410036351678@bro-tracker.atlassian.net>
Message-ID: <JIRA.15802.1410036351000.19.1410036367649@Atlassian.JIRA>

Jimmy Jones created BIT-1240:
--------------------------------

             Summary: TCP gaps inserted in wrong place
                 Key: BIT-1240
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1240
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: 2.3, git/master
         Environment: CentOS 6
            Reporter: Jimmy Jones
         Attachments: get-hole1.trace

Using attached test file, I tried using the file analysis framework to extract out the payload, which is a copy of one of the bro testcases but with a packet removed.

However the extracted file has nulls padded at the beginning, but they should be at offset 1448. Looking at what happens, the gap is signalled to File::Gap before the data is received in File::DataIn, which calculates the offset to write the payload by adding the seen bytes to the missing byte count. Should gaps always be signalled in order with the data - this also affects users of content_gap, who would receive the data and hole "out of order"?

Used the following bro script:

event file_new(f: fa_file)
{
  Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]);
}



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Sat Sep  6 18:47:07 2014
From: jira at bro-tracker.atlassian.net (gclark (JIRA))
Date: Sat, 6 Sep 2014 20:47:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1241) Plugin: HandlePluginResult fix for
	TYPE_ANY
In-Reply-To: <JIRA.15803.1410054375000@Atlassian.JIRA>
References: <JIRA.15803.1410054375000@Atlassian.JIRA>
	<JIRA.15803.1410054375279@bro-tracker.atlassian.net>
Message-ID: <JIRA.15803.1410054375000.20.1410054427647@Atlassian.JIRA>

gclark created BIT-1241:
---------------------------

             Summary: Plugin: HandlePluginResult fix for TYPE_ANY
                 Key: BIT-1241
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1241
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: 2.3
            Reporter: gclark
            Assignee: gclark
            Priority: Low


We should disable strict type sanity checking in HandlePluginResult for bro functions that return TYPE_ANY.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Sat Sep  6 18:50:07 2014
From: jira at bro-tracker.atlassian.net (gclark (JIRA))
Date: Sat, 6 Sep 2014 20:50:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1241) Plugin: HandlePluginResult fix for
	TYPE_ANY
In-Reply-To: <JIRA.15803.1410054375000@Atlassian.JIRA>
References: <JIRA.15803.1410054375000@Atlassian.JIRA>
	<JIRA.15803.1410054375279@bro-tracker.atlassian.net>
Message-ID: <JIRA.15803.1410054375000.21.1410054607658@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

gclark updated BIT-1241:
------------------------
    Description: 
We should disable strict type sanity checking in HandlePluginResult for bro functions that return TYPE_ANY.

This is needed to wrap functions such as Queue::get() where the function is declared with a return type of 'any', but often returns items that are of a specific type (e.g. record).  Although a returned record value is valid, the type sanity check in Func::HandlePluginResult fails because TYPE_RECORD != TYPE_ANY.

  was:We should disable strict type sanity checking in HandlePluginResult for bro functions that return TYPE_ANY.


> Plugin: HandlePluginResult fix for TYPE_ANY
> -------------------------------------------
>
>                 Key: BIT-1241
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1241
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>            Reporter: gclark
>            Assignee: gclark
>            Priority: Low
>
> We should disable strict type sanity checking in HandlePluginResult for bro functions that return TYPE_ANY.
> This is needed to wrap functions such as Queue::get() where the function is declared with a return type of 'any', but often returns items that are of a specific type (e.g. record).  Although a returned record value is valid, the type sanity check in Func::HandlePluginResult fails because TYPE_RECORD != TYPE_ANY.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Sat Sep  6 18:50:07 2014
From: jira at bro-tracker.atlassian.net (gclark (JIRA))
Date: Sat, 6 Sep 2014 20:50:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1241) Plugin: HandlePluginResult fix for
	TYPE_ANY
In-Reply-To: <JIRA.15803.1410054375000@Atlassian.JIRA>
References: <JIRA.15803.1410054375000@Atlassian.JIRA>
	<JIRA.15803.1410054375279@bro-tracker.atlassian.net>
Message-ID: <JIRA.15803.1410054375000.22.1410054607710@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

gclark updated BIT-1241:
------------------------
    Affects Version/s:     (was: 2.3)

> Plugin: HandlePluginResult fix for TYPE_ANY
> -------------------------------------------
>
>                 Key: BIT-1241
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1241
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>            Reporter: gclark
>            Assignee: gclark
>            Priority: Low
>
> We should disable strict type sanity checking in HandlePluginResult for bro functions that return TYPE_ANY.
> This is needed to wrap functions such as Queue::get() where the function is declared with a return type of 'any', but often returns items that are of a specific type (e.g. record).  Although a returned record value is valid, the type sanity check in Func::HandlePluginResult fails because TYPE_RECORD != TYPE_ANY.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From noreply at bro.org  Sun Sep  7 00:00:22 2014
From: noreply at bro.org (Merge Tracker)
Date: Sun, 7 Sep 2014 00:00:22 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409070700.s8770MQ2002778@bro-ids.icir.org>


Open Merge Requests
===================

ID            Component    Reporter    Assignee      Updated     For Version    Priority    Summary
------------  -----------  ----------  ------------  ----------  -------------  ----------  ----------------------------------------
BIT-1231 [1]  Bro          hui         Robin Sommer  2014-09-04  -              Normal      DNP3 Analyzer Supports for DNP3-over-UDP


Open Fastpath Commits
======================

Commit       Component    Author         Date        Summary
-----------  -----------  -------------  ----------  --------------------
f01e862 [2]  bro          Johanna Amann  2014-09-04  fix more http links.


[1]  BIT-1231  https://bro-tracker.atlassian.net/browse/BIT-1231
[2]  f01e862   https://github.com/bro/bro/commit/f01e8629fcb6acec5001bc2f55a396e174228a9a


From jira at bro-tracker.atlassian.net  Sun Sep  7 21:40:08 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Sun, 7 Sep 2014 23:40:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1231) DNP3 Analyzer Supports for DNP3-over-UDP
In-Reply-To: <JIRA.15400.1408049217000@Atlassian.JIRA>
References: <JIRA.15400.1408049217000@Atlassian.JIRA>
	<JIRA.15400.1408049217421@bro-tracker.atlassian.net>
Message-ID: <JIRA.15400.1408049217000.2.1410151208220@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robin Sommer updated BIT-1231:
------------------------------
    Status: Open  (was: Merge Request)

> DNP3 Analyzer Supports for DNP3-over-UDP
> ----------------------------------------
>
>                 Key: BIT-1231
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1231
>             Project: Bro Issue Tracker
>          Issue Type: Improvement
>          Components: Bro
>    Affects Versions: 2.3
>            Reporter: hui
>            Assignee: Robin Sommer
>              Labels: DNP3, analyzer
>
> Two major changes are made for the DNP3 analyzer
> 1. Make the analyzer support both the DNP3-over-UDP and the DNP3-over-TCP.
>     The changes are made in DNP3.cc, DNP3.h and dpd.sig
> 2. Fix a bug in the binpac codes of the DNP3 analyzer
>    The changes are made in dnp3-protocol.pac. The changes results in different baseline results of testing/btest/Baseline/scripts.base.protocols.dnp3.dnp3_link_only
>  



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Sun Sep  7 21:41:07 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Sun, 7 Sep 2014 23:41:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1231) DNP3 Analyzer Supports for DNP3-over-UDP
In-Reply-To: <JIRA.15400.1408049217000@Atlassian.JIRA>
References: <JIRA.15400.1408049217000@Atlassian.JIRA>
	<JIRA.15400.1408049217421@bro-tracker.atlassian.net>
Message-ID: <JIRA.15400.1408049217000.4.1410151267688@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robin Sommer reassigned BIT-1231:
---------------------------------

    Assignee: hui  (was: Robin Sommer)

Much better, thanks!

I've tweaked it a little bit more, see topic/robin/dnp3-merge-v4.

Now just one more request: I don't see any test case for DNP3 over UDP. Can you  add one  please? (adding to your branch is fine).

> DNP3 Analyzer Supports for DNP3-over-UDP
> ----------------------------------------
>
>                 Key: BIT-1231
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1231
>             Project: Bro Issue Tracker
>          Issue Type: Improvement
>          Components: Bro
>    Affects Versions: 2.3
>            Reporter: hui
>            Assignee: hui
>              Labels: DNP3, analyzer
>
> Two major changes are made for the DNP3 analyzer
> 1. Make the analyzer support both the DNP3-over-UDP and the DNP3-over-TCP.
>     The changes are made in DNP3.cc, DNP3.h and dpd.sig
> 2. Fix a bug in the binpac codes of the DNP3 analyzer
>    The changes are made in dnp3-protocol.pac. The changes results in different baseline results of testing/btest/Baseline/scripts.base.protocols.dnp3.dnp3_link_only
>  



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Mon Sep  8 11:58:07 2014
From: jira at bro-tracker.atlassian.net (Jeannette Dopheide (JIRA))
Date: Mon, 8 Sep 2014 13:58:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1242) Logs: add page listing logs to
 /script-reference/index.html
In-Reply-To: <JIRA.15900.1410202655000@Atlassian.JIRA>
References: <JIRA.15900.1410202655000@Atlassian.JIRA>
	<JIRA.15900.1410202655434@bro-tracker.atlassian.net>
Message-ID: <JIRA.15900.1410202655000.5.1410202687648@Atlassian.JIRA>

Jeannette Dopheide created BIT-1242:
---------------------------------------

             Summary: Logs: add page listing logs to /script-reference/index.html
                 Key: BIT-1242
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1242
             Project: Bro Issue Tracker
          Issue Type: Improvement
          Components: Documentation, Website
            Reporter: Jeannette Dopheide
            Assignee: Jeannette Dopheide


Add a page to documentation listing the Bro log files.

There's already a page listing common log files: 
http://www.bro.org/sphinx/logs/index.html#common-log-files

This page could be moved to http://www.bro.org/sphinx/script-reference/index.html and be expanded to list all known log files.

If this goes well, Robin suggested adding a btest that heuristically checks if something changes with regards to which logs we have.  I'll open a ticket for that when we're done.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Mon Sep  8 13:43:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Mon, 8 Sep 2014 15:43:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1240) TCP gaps inserted in wrong place
In-Reply-To: <JIRA.15802.1410036351000@Atlassian.JIRA>
References: <JIRA.15802.1410036351000@Atlassian.JIRA>
	<JIRA.15802.1410036351678@bro-tracker.atlassian.net>
Message-ID: <JIRA.15802.1410036351000.7.1410208987761@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18001#comment-18001 ] 

Jon Siwek commented on BIT-1240:
--------------------------------

{quote}
However the extracted file has nulls padded at the beginning, but they should be at offset 1448.
{quote}

Or rather the gap should start at byte offset 1146, I think?  The first packet w/ reply content has 1448 bytes of data, but 302 are HTTP-specific stuff (e.g. headers), not related to the "file".  So 1448 - 302 = 1146.

{quote}
Looking at what happens, the gap is signalled to File::Gap before the data is received in File::DataIn, which calculates the offset to write the payload by adding the seen bytes to the missing byte count.  Should gaps always be signalled in order with the data 
{quote}

Yes, that is how it should work, thanks for the bug report.

{quote}
this also affects users of content_gap, who would receive the data and hole "out of order"?
{quote}

If you literally mean the "content_gap" event, I'm guessing that's unaffected as the issue isn't related to TCP reassembly/analysis.  The source of the problem is specifically w/ how MIME entity data is buffered and passed along to protocol analyzers (and then to file analyzers) in discrete amounts, but gaps are always reported right away.  So either gaps need to be buffered along w/ the data, or the buffering needs to be removed.  I'll see if I can do the later as that may have additional benefits of decoupling HTTP file analysis from some things it's not supposed to be coupled with...

> TCP gaps inserted in wrong place
> --------------------------------
>
>                 Key: BIT-1240
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1240
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: get-hole1.trace
>
>
> Using attached test file, I tried using the file analysis framework to extract out the payload, which is a copy of one of the bro testcases but with a packet removed.
> However the extracted file has nulls padded at the beginning, but they should be at offset 1448. Looking at what happens, the gap is signalled to File::Gap before the data is received in File::DataIn, which calculates the offset to write the payload by adding the seen bytes to the missing byte count. Should gaps always be signalled in order with the data - this also affects users of content_gap, who would receive the data and hole "out of order"?
> Used the following bro script:
> event file_new(f: fa_file)
> {
>   Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]);
> }



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Mon Sep  8 14:12:07 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Mon, 8 Sep 2014 16:12:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1243) topic/robin/pktsrc
In-Reply-To: <JIRA.15901.1410210699000@Atlassian.JIRA>
References: <JIRA.15901.1410210699000@Atlassian.JIRA>
	<JIRA.15901.1410210699950@bro-tracker.atlassian.net>
Message-ID: <JIRA.15901.1410210699000.8.1410210727635@Atlassian.JIRA>

Robin Sommer created BIT-1243:
---------------------------------

             Summary: topic/robin/pktsrc
                 Key: BIT-1243
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1243
             Project: Bro Issue Tracker
          Issue Type: New Feature
          Components: Bro
            Reporter: Robin Sommer


This moves packet sources and dumpers over to the new dynamic plugin infrastructure. 

By default, Bro still supports just pcap for either, but other formats can now be added externally as well. A user choses a different format by specifying  a corresponding prefix with the -r or -i options (e.g., "-i netmap:eth0") While the builtin pcap supports remains the default, one can select it explicitly via its "pcap" prefix (e.g., "-r pcap:my.trace").

This branch is in bro, bro-aux, and cmake. 

Further changes included:

- Removal of the "secondary path".
- Removal of NeFflow support. We can bring this back, but need to decide where it should hook in now.
- Various smaller tweaks to the plugin infrastructure and skeleton.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Mon Sep  8 14:12:07 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Mon, 8 Sep 2014 16:12:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1243) topic/robin/pktsrc
In-Reply-To: <JIRA.15901.1410210699000@Atlassian.JIRA>
References: <JIRA.15901.1410210699000@Atlassian.JIRA>
	<JIRA.15901.1410210699950@bro-tracker.atlassian.net>
Message-ID: <JIRA.15901.1410210699000.9.1410210727691@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robin Sommer updated BIT-1243:
------------------------------
    Status: Merge Request  (was: Open)

> topic/robin/pktsrc
> ------------------
>
>                 Key: BIT-1243
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1243
>             Project: Bro Issue Tracker
>          Issue Type: New Feature
>          Components: Bro
>            Reporter: Robin Sommer
>
> This moves packet sources and dumpers over to the new dynamic plugin infrastructure. 
> By default, Bro still supports just pcap for either, but other formats can now be added externally as well. A user choses a different format by specifying  a corresponding prefix with the -r or -i options (e.g., "-i netmap:eth0") While the builtin pcap supports remains the default, one can select it explicitly via its "pcap" prefix (e.g., "-r pcap:my.trace").
> This branch is in bro, bro-aux, and cmake. 
> Further changes included:
> - Removal of the "secondary path".
> - Removal of NeFflow support. We can bring this back, but need to decide where it should hook in now.
> - Various smaller tweaks to the plugin infrastructure and skeleton.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Mon Sep  8 14:16:07 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Mon, 8 Sep 2014 16:16:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1244) topic/robin/netmap
In-Reply-To: <JIRA.15902.1410210947000@Atlassian.JIRA>
References: <JIRA.15902.1410210947000@Atlassian.JIRA>
	<JIRA.15902.1410210947999@bro-tracker.atlassian.net>
Message-ID: <JIRA.15902.1410210947000.10.1410210967632@Atlassian.JIRA>

Robin Sommer created BIT-1244:
---------------------------------

             Summary: topic/robin/netmap
                 Key: BIT-1244
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1244
             Project: Bro Issue Tracker
          Issue Type: New Feature
          Components: Bro
            Reporter: Robin Sommer


This branch adds native netmap support as an external plugin. Once installed, a user can activate it by using, e.g., "-i netmap:eth0". See the README for more information.

This needs BIT-1243 merged first.

Tested on Linux only right now (and not very much).



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Mon Sep  8 14:16:07 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Mon, 8 Sep 2014 16:16:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1244) topic/robin/netmap
In-Reply-To: <JIRA.15902.1410210947000@Atlassian.JIRA>
References: <JIRA.15902.1410210947000@Atlassian.JIRA>
	<JIRA.15902.1410210947999@bro-tracker.atlassian.net>
Message-ID: <JIRA.15902.1410210947000.11.1410210967684@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robin Sommer updated BIT-1244:
------------------------------
    Status: Merge Request  (was: Open)

> topic/robin/netmap
> ------------------
>
>                 Key: BIT-1244
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1244
>             Project: Bro Issue Tracker
>          Issue Type: New Feature
>          Components: Bro
>            Reporter: Robin Sommer
>
> This branch adds native netmap support as an external plugin. Once installed, a user can activate it by using, e.g., "-i netmap:eth0". See the README for more information.
> This needs BIT-1243 merged first.
> Tested on Linux only right now (and not very much).



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Mon Sep  8 16:22:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Mon, 8 Sep 2014 18:22:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1240) TCP gaps inserted in wrong place
In-Reply-To: <JIRA.15802.1410036351000@Atlassian.JIRA>
References: <JIRA.15802.1410036351000@Atlassian.JIRA>
	<JIRA.15802.1410036351678@bro-tracker.atlassian.net>
Message-ID: <JIRA.15802.1410036351000.13.1410218527709@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18002#comment-18002 ] 

Jon Siwek commented on BIT-1240:
--------------------------------

There's a "topic/jsiwek/bit-1240" branch now w/ a change that should help if you want to check it out and let me know.

> TCP gaps inserted in wrong place
> --------------------------------
>
>                 Key: BIT-1240
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1240
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: get-hole1.trace
>
>
> Using attached test file, I tried using the file analysis framework to extract out the payload, which is a copy of one of the bro testcases but with a packet removed.
> However the extracted file has nulls padded at the beginning, but they should be at offset 1448. Looking at what happens, the gap is signalled to File::Gap before the data is received in File::DataIn, which calculates the offset to write the payload by adding the seen bytes to the missing byte count. Should gaps always be signalled in order with the data - this also affects users of content_gap, who would receive the data and hole "out of order"?
> Used the following bro script:
> event file_new(f: fa_file)
> {
>   Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]);
> }



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Mon Sep  8 16:23:08 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Mon, 8 Sep 2014 18:23:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1240) TCP gaps inserted in wrong place
In-Reply-To: <JIRA.15802.1410036351000@Atlassian.JIRA>
References: <JIRA.15802.1410036351000@Atlassian.JIRA>
	<JIRA.15802.1410036351678@bro-tracker.atlassian.net>
Message-ID: <JIRA.15802.1410036351000.15.1410218588353@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1240:
---------------------------
    Fix Version/s: 2.4

> TCP gaps inserted in wrong place
> --------------------------------
>
>                 Key: BIT-1240
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1240
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>             Fix For: 2.4
>
>         Attachments: get-hole1.trace
>
>
> Using attached test file, I tried using the file analysis framework to extract out the payload, which is a copy of one of the bro testcases but with a packet removed.
> However the extracted file has nulls padded at the beginning, but they should be at offset 1448. Looking at what happens, the gap is signalled to File::Gap before the data is received in File::DataIn, which calculates the offset to write the payload by adding the seen bytes to the missing byte count. Should gaps always be signalled in order with the data - this also affects users of content_gap, who would receive the data and hole "out of order"?
> Used the following bro script:
> event file_new(f: fa_file)
> {
>   Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]);
> }



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Mon Sep  8 16:36:07 2014
From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA))
Date: Mon, 8 Sep 2014 18:36:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1245) Opaque types do not get displayed
 correctly in generated documentation
In-Reply-To: <JIRA.15903.1410219365000@Atlassian.JIRA>
References: <JIRA.15903.1410219365000@Atlassian.JIRA>
	<JIRA.15903.1410219365374@bro-tracker.atlassian.net>
Message-ID: <JIRA.15903.1410219365000.16.1410219367624@Atlassian.JIRA>

Johanna Amann created BIT-1245:
----------------------------------

             Summary: Opaque types do not get displayed correctly in generated documentation
                 Key: BIT-1245
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1245
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Documentation
    Affects Versions: git/master
            Reporter: Johanna Amann


Opaque types do not get their full types displayed in the Documentation.

For example, im https://www.bro.org/sphinx-git/scripts/base/bif/plugins/Bro_X509.events.bif.bro.html x509_certificate has a signature of ..., cert_ref: opaque, ... instead of the correct ..., cert_ref: opaque of x509, ...

Bug was discovered by Michal Purzynski



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Mon Sep  8 16:37:07 2014
From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA))
Date: Mon, 8 Sep 2014 18:37:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1245) Opaque types do not get displayed
 correctly in generated documentation
In-Reply-To: <JIRA.15903.1410219365000@Atlassian.JIRA>
References: <JIRA.15903.1410219365000@Atlassian.JIRA>
	<JIRA.15903.1410219365374@bro-tracker.atlassian.net>
Message-ID: <JIRA.15903.1410219365000.17.1410219427627@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Johanna Amann updated BIT-1245:
-------------------------------
    Fix Version/s: 2.4

> Opaque types do not get displayed correctly in generated documentation
> ----------------------------------------------------------------------
>
>                 Key: BIT-1245
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1245
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Documentation
>    Affects Versions: git/master
>            Reporter: Johanna Amann
>             Fix For: 2.4
>
>
> Opaque types do not get their full types displayed in the Documentation.
> For example, im https://www.bro.org/sphinx-git/scripts/base/bif/plugins/Bro_X509.events.bif.bro.html x509_certificate has a signature of ..., cert_ref: opaque, ... instead of the correct ..., cert_ref: opaque of x509, ...
> Bug was discovered by Michal Purzynski



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Mon Sep  8 17:06:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Mon, 8 Sep 2014 19:06:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1245) Opaque types do not get displayed
 correctly in generated documentation
In-Reply-To: <JIRA.15903.1410219365000@Atlassian.JIRA>
References: <JIRA.15903.1410219365000@Atlassian.JIRA>
	<JIRA.15903.1410219365374@bro-tracker.atlassian.net>
Message-ID: <JIRA.15903.1410219365000.19.1410221167682@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1245:
---------------------------
    Resolution: Fixed
        Status: Closed  (was: Open)

> Opaque types do not get displayed correctly in generated documentation
> ----------------------------------------------------------------------
>
>                 Key: BIT-1245
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1245
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Documentation
>    Affects Versions: git/master
>            Reporter: Johanna Amann
>             Fix For: 2.4
>
>
> Opaque types do not get their full types displayed in the Documentation.
> For example, im https://www.bro.org/sphinx-git/scripts/base/bif/plugins/Bro_X509.events.bif.bro.html x509_certificate has a signature of ..., cert_ref: opaque, ... instead of the correct ..., cert_ref: opaque of x509, ...
> Bug was discovered by Michal Purzynski



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From noreply at bro.org  Tue Sep  9 00:00:21 2014
From: noreply at bro.org (Merge Tracker)
Date: Tue, 9 Sep 2014 00:00:21 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409090700.s8970LRx006128@bro-ids.icir.org>


Open Merge Requests
===================

ID            Component    Reporter      Assignee    Updated     For Version    Priority    Summary
------------  -----------  ------------  ----------  ----------  -------------  ----------  ----------------------
BIT-1244 [1]  Bro          Robin Sommer  -           2014-09-08  -              Normal      topic/robin/netmap [2]
BIT-1243 [3]  Bro          Robin Sommer  -           2014-09-08  -              Normal      topic/robin/pktsrc [4]


[1]  BIT-1244  https://bro-tracker.atlassian.net/browse/BIT-1244
[2]  netmap    https://github.com/bro/bro/tree/topic/robin/netmap
[3]  BIT-1243  https://bro-tracker.atlassian.net/browse/BIT-1243
[4]  pktsrc    https://github.com/bro/bro/tree/topic/robin/pktsrc


From jira at bro-tracker.atlassian.net  Tue Sep  9 07:35:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Tue, 9 Sep 2014 09:35:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1243) topic/robin/pktsrc
In-Reply-To: <JIRA.15901.1410210699000@Atlassian.JIRA>
References: <JIRA.15901.1410210699000@Atlassian.JIRA>
	<JIRA.15901.1410210699950@bro-tracker.atlassian.net>
Message-ID: <JIRA.15901.1410210699000.21.1410273307727@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18003#comment-18003 ] 

Jon Siwek commented on BIT-1243:
--------------------------------

Is this branch pushed to bro-aux and cmake ?

> topic/robin/pktsrc
> ------------------
>
>                 Key: BIT-1243
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1243
>             Project: Bro Issue Tracker
>          Issue Type: New Feature
>          Components: Bro
>            Reporter: Robin Sommer
>
> This moves packet sources and dumpers over to the new dynamic plugin infrastructure. 
> By default, Bro still supports just pcap for either, but other formats can now be added externally as well. A user choses a different format by specifying  a corresponding prefix with the -r or -i options (e.g., "-i netmap:eth0") While the builtin pcap supports remains the default, one can select it explicitly via its "pcap" prefix (e.g., "-r pcap:my.trace").
> This branch is in bro, bro-aux, and cmake. 
> Further changes included:
> - Removal of the "secondary path".
> - Removal of NeFflow support. We can bring this back, but need to decide where it should hook in now.
> - Various smaller tweaks to the plugin infrastructure and skeleton.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Tue Sep  9 07:37:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Tue, 9 Sep 2014 09:37:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1243) topic/robin/pktsrc
In-Reply-To: <JIRA.15901.1410210699000@Atlassian.JIRA>
References: <JIRA.15901.1410210699000@Atlassian.JIRA>
	<JIRA.15901.1410210699950@bro-tracker.atlassian.net>
Message-ID: <JIRA.15901.1410210699000.23.1410273427726@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18003#comment-18003 ] 

Jon Siwek edited comment on BIT-1243 at 9/9/14 9:36 AM:
--------------------------------------------------------

Is this branch pushed to bro-aux and cmake ?  Not finding it and the submodules were referring to nonexisting commits.


was (Author: jsiwek):
Is this branch pushed to bro-aux and cmake ?

> topic/robin/pktsrc
> ------------------
>
>                 Key: BIT-1243
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1243
>             Project: Bro Issue Tracker
>          Issue Type: New Feature
>          Components: Bro
>            Reporter: Robin Sommer
>
> This moves packet sources and dumpers over to the new dynamic plugin infrastructure. 
> By default, Bro still supports just pcap for either, but other formats can now be added externally as well. A user choses a different format by specifying  a corresponding prefix with the -r or -i options (e.g., "-i netmap:eth0") While the builtin pcap supports remains the default, one can select it explicitly via its "pcap" prefix (e.g., "-r pcap:my.trace").
> This branch is in bro, bro-aux, and cmake. 
> Further changes included:
> - Removal of the "secondary path".
> - Removal of NeFflow support. We can bring this back, but need to decide where it should hook in now.
> - Various smaller tweaks to the plugin infrastructure and skeleton.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Tue Sep  9 07:38:07 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Tue, 9 Sep 2014 09:38:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1243) topic/robin/pktsrc
In-Reply-To: <JIRA.15901.1410210699000@Atlassian.JIRA>
References: <JIRA.15901.1410210699000@Atlassian.JIRA>
	<JIRA.15901.1410210699950@bro-tracker.atlassian.net>
Message-ID: <JIRA.15901.1410210699000.24.1410273487625@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18004#comment-18004 ] 

Robin Sommer commented on BIT-1243:
-----------------------------------

Now it is.






> topic/robin/pktsrc
> ------------------
>
>                 Key: BIT-1243
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1243
>             Project: Bro Issue Tracker
>          Issue Type: New Feature
>          Components: Bro
>            Reporter: Robin Sommer
>
> This moves packet sources and dumpers over to the new dynamic plugin infrastructure. 
> By default, Bro still supports just pcap for either, but other formats can now be added externally as well. A user choses a different format by specifying  a corresponding prefix with the -r or -i options (e.g., "-i netmap:eth0") While the builtin pcap supports remains the default, one can select it explicitly via its "pcap" prefix (e.g., "-r pcap:my.trace").
> This branch is in bro, bro-aux, and cmake. 
> Further changes included:
> - Removal of the "secondary path".
> - Removal of NeFflow support. We can bring this back, but need to decide where it should hook in now.
> - Various smaller tweaks to the plugin infrastructure and skeleton.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Tue Sep  9 11:00:08 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Tue, 9 Sep 2014 13:00:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1243) topic/robin/pktsrc
In-Reply-To: <JIRA.15901.1410210699000@Atlassian.JIRA>
References: <JIRA.15901.1410210699000@Atlassian.JIRA>
	<JIRA.15901.1410210699950@bro-tracker.atlassian.net>
Message-ID: <JIRA.15901.1410210699000.26.1410285608090@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1243:
---------------------------
    Fix Version/s: 2.4
           Status: Closed  (was: Merge Request)

I tried to condense some things in CHANGES, please edit if you'd like to change/add anything.

> topic/robin/pktsrc
> ------------------
>
>                 Key: BIT-1243
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1243
>             Project: Bro Issue Tracker
>          Issue Type: New Feature
>          Components: Bro
>            Reporter: Robin Sommer
>             Fix For: 2.4
>
>
> This moves packet sources and dumpers over to the new dynamic plugin infrastructure. 
> By default, Bro still supports just pcap for either, but other formats can now be added externally as well. A user choses a different format by specifying  a corresponding prefix with the -r or -i options (e.g., "-i netmap:eth0") While the builtin pcap supports remains the default, one can select it explicitly via its "pcap" prefix (e.g., "-r pcap:my.trace").
> This branch is in bro, bro-aux, and cmake. 
> Further changes included:
> - Removal of the "secondary path".
> - Removal of NeFflow support. We can bring this back, but need to decide where it should hook in now.
> - Various smaller tweaks to the plugin infrastructure and skeleton.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Tue Sep  9 11:18:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Tue, 9 Sep 2014 13:18:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1244) topic/robin/netmap
In-Reply-To: <JIRA.15902.1410210947000@Atlassian.JIRA>
References: <JIRA.15902.1410210947000@Atlassian.JIRA>
	<JIRA.15902.1410210947999@bro-tracker.atlassian.net>
Message-ID: <JIRA.15902.1410210947000.28.1410286687683@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1244:
---------------------------
    Resolution: Merged  (was: Fixed)
        Status: Closed  (was: Merge Request)

> topic/robin/netmap
> ------------------
>
>                 Key: BIT-1244
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1244
>             Project: Bro Issue Tracker
>          Issue Type: New Feature
>          Components: Bro
>            Reporter: Robin Sommer
>
> This branch adds native netmap support as an external plugin. Once installed, a user can activate it by using, e.g., "-i netmap:eth0". See the README for more information.
> This needs BIT-1243 merged first.
> Tested on Linux only right now (and not very much).



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Tue Sep  9 11:19:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Tue, 9 Sep 2014 13:19:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1244) topic/robin/netmap
In-Reply-To: <JIRA.15902.1410210947000@Atlassian.JIRA>
References: <JIRA.15902.1410210947000@Atlassian.JIRA>
	<JIRA.15902.1410210947999@bro-tracker.atlassian.net>
Message-ID: <JIRA.15902.1410210947000.30.1410286747642@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1244:
---------------------------
    Fix Version/s: 2.4

> topic/robin/netmap
> ------------------
>
>                 Key: BIT-1244
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1244
>             Project: Bro Issue Tracker
>          Issue Type: New Feature
>          Components: Bro
>            Reporter: Robin Sommer
>             Fix For: 2.4
>
>
> This branch adds native netmap support as an external plugin. Once installed, a user can activate it by using, e.g., "-i netmap:eth0". See the README for more information.
> This needs BIT-1243 merged first.
> Tested on Linux only right now (and not very much).



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Tue Sep  9 12:58:07 2014
From: jira at bro-tracker.atlassian.net (Jeannette Dopheide (JIRA))
Date: Tue, 9 Sep 2014 14:58:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1242) Logs: add page listing logs to
 /script-reference/index.html
In-Reply-To: <JIRA.15900.1410202655000@Atlassian.JIRA>
References: <JIRA.15900.1410202655000@Atlassian.JIRA>
	<JIRA.15900.1410202655434@bro-tracker.atlassian.net>
Message-ID: <JIRA.15900.1410202655000.31.1410292687655@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeannette Dopheide updated BIT-1242:
------------------------------------
    Status: In Progress  (was: Open)

> Logs: add page listing logs to /script-reference/index.html
> -----------------------------------------------------------
>
>                 Key: BIT-1242
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1242
>             Project: Bro Issue Tracker
>          Issue Type: Improvement
>          Components: Documentation, Website
>            Reporter: Jeannette Dopheide
>            Assignee: Jeannette Dopheide
>
> Add a page to documentation listing the Bro log files.
> There's already a page listing common log files: 
> http://www.bro.org/sphinx/logs/index.html#common-log-files
> This page could be moved to http://www.bro.org/sphinx/script-reference/index.html and be expanded to list all known log files.
> If this goes well, Robin suggested adding a btest that heuristically checks if something changes with regards to which logs we have.  I'll open a ticket for that when we're done.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Tue Sep  9 13:22:07 2014
From: jira at bro-tracker.atlassian.net (Brian O'Berry (JIRA))
Date: Tue, 9 Sep 2014 15:22:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1235) HTTP multipart POST request alters file
	contents
In-Reply-To: <JIRA.15503.1408906826000@Atlassian.JIRA>
References: <JIRA.15503.1408906826000@Atlassian.JIRA>
	<JIRA.15503.1408906826671@bro-tracker.atlassian.net>
Message-ID: <JIRA.15503.1408906826000.32.1410294127628@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18006#comment-18006 ] 

Brian O'Berry commented on BIT-1235:
------------------------------------

Hi, Jon,

A quick update... I made the change as documented in BIT-1235, and we've been running it in our unit test environment since then.  All feedback is positive so far.  Our release cycle got extended to add a couple significant features, so the fix won't go into QA testing until the week after next.  If things go according to the new schedule, we'll have it running on our corporate network a week after that, where we'll watch it for another week or so.  I anticipate updating the ticket for the final time in 3 weeks or so.

Thanks again for the quick fix, and let me know if you need anything.

Best,

Brian



> HTTP multipart POST request alters file contents
> ------------------------------------------------
>
>                 Key: BIT-1235
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1235
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: 2.3
>         Environment: CentOS 6.5, file extract analyzer
>            Reporter: Brian O'Berry
>             Fix For: 2.4
>
>         Attachments: bro-2.3-HTTP.patch, gdb.log, upload-api-http.pcap
>
>
> HTTP POST multipart processing converts bare CR or LF chars to CRLF pairs, corrupting most files when extracted with Files::ANALYZER_EXTRACT.  This is clear in the attached gdb.log, which has a backtrace that shows a buffer with the start of a PDF file entering MIME/HTTP entity processing at frame 25, and emerging with LF chars converted to CRLF at frame 6.
> Also attached are the pcap file associated with the backtrace, and an initial patch that we've barely begun to test.  A point of concern with the patch is that it changes a weird.log entry from "line_terminated_with_single_CR" to "http_no_crlf_in_header_list".  It does enable Files::ANALYZER_EXTRACT to correctly extract the PDF file from the attached pcap.
> Please let me know if we can provide anything else to help with this.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From noreply at bro.org  Wed Sep 10 00:00:21 2014
From: noreply at bro.org (Merge Tracker)
Date: Wed, 10 Sep 2014 00:00:21 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409100700.s8A70LHK024132@bro-ids.icir.org>


Open GitHub Pull Requests
=========================

Issue    Component    User        Updated     Title
-------  -----------  ----------  ----------  ------------------------
#1 [1]   bro-scripts  cmavr8 [2]  2014-09-09  Fixed typo in README [3]


[1]  Pull Request #1             https://github.com/bro/bro-scripts/pull/1
[2]  cmavr8                      https://github.com/cmavr8
[3]  Merge Pull Request #1 with  git pull --no-ff --no-commit https://github.com/cmavr8/bro-scripts.git master


From jira at bro-tracker.atlassian.net  Wed Sep 10 08:02:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Wed, 10 Sep 2014 10:02:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1240) TCP gaps inserted in wrong place
In-Reply-To: <JIRA.15802.1410036351000@Atlassian.JIRA>
References: <JIRA.15802.1410036351000@Atlassian.JIRA>
	<JIRA.15802.1410036351678@bro-tracker.atlassian.net>
Message-ID: <JIRA.15802.1410036351000.33.1410361327657@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18007#comment-18007 ] 

Jimmy Jones commented on BIT-1240:
----------------------------------

Yes, my math was a bit off! Have tested it with my samples and it works great, thanks!

My use case, where I stumbled across the issue, was where I was receiving http_entity_data events and also content_gap events so I could reconstruct a POST or response, and punch the holes in the right place. However the content_gap event didn't occur "in order" with http_entity_data (in this case before the content_gap occurred before all of the http_entity_data). Would you like me to raise a separate issue for this, or keep it here?

> TCP gaps inserted in wrong place
> --------------------------------
>
>                 Key: BIT-1240
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1240
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>             Fix For: 2.4
>
>         Attachments: get-hole1.trace
>
>
> Using attached test file, I tried using the file analysis framework to extract out the payload, which is a copy of one of the bro testcases but with a packet removed.
> However the extracted file has nulls padded at the beginning, but they should be at offset 1448. Looking at what happens, the gap is signalled to File::Gap before the data is received in File::DataIn, which calculates the offset to write the payload by adding the seen bytes to the missing byte count. Should gaps always be signalled in order with the data - this also affects users of content_gap, who would receive the data and hole "out of order"?
> Used the following bro script:
> event file_new(f: fa_file)
> {
>   Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]);
> }



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Wed Sep 10 08:09:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Wed, 10 Sep 2014 10:09:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1246) Missing last TCP segment
In-Reply-To: <JIRA.15904.1410361694000@Atlassian.JIRA>
References: <JIRA.15904.1410361694000@Atlassian.JIRA>
	<JIRA.15904.1410361694392@bro-tracker.atlassian.net>
Message-ID: <JIRA.15904.1410361694000.34.1410361747624@Atlassian.JIRA>

Jimmy Jones created BIT-1246:
--------------------------------

             Summary: Missing last TCP segment
                 Key: BIT-1246
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1246
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: 2.3, git/master
         Environment: CentOS 6
            Reporter: Jimmy Jones
         Attachments: get-hole2.trace

Possibly related to BIT-1240, I've got a sample (attached) where the penultimate TCP segment of the http response is missing.

Using the file analysis framework I'm extracting the payload, but I don't get any of the last segment. Without the fix in BIT-1240 additionally the NULL padding for the missing packets happens in the wrong place.

Used the following bro script:

event file_new(f: fa_file)
{ Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]); } 



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Wed Sep 10 08:42:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Wed, 10 Sep 2014 10:42:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1247) Missing packet in HTTP byte ranges
 request stops processing
In-Reply-To: <JIRA.15905.1410363677000@Atlassian.JIRA>
References: <JIRA.15905.1410363677000@Atlassian.JIRA>
	<JIRA.15905.1410363677334@bro-tracker.atlassian.net>
Message-ID: <JIRA.15905.1410363677000.35.1410363727647@Atlassian.JIRA>

Jimmy Jones created BIT-1247:
--------------------------------

             Summary: Missing packet in HTTP byte ranges request stops processing
                 Key: BIT-1247
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1247
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: git/master
         Environment: CentOS 6
            Reporter: Jimmy Jones
         Attachments: byteranges-hole1.trace

I've created the attached file from one in the testing framework, but with packet #8 removed. The missing packet is in the middle of a mime part and doesn't straddle any MIME boundaries. However with the packet removed, the file output by the file analysis framework only contains the data up until the missing packet. As the missing packet didn't include any MIME boundaries, I wouldn't expect this behavior.

Used the following bro script:

event file_new(f: fa_file)
{ Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]); } 



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Wed Sep 10 09:02:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Wed, 10 Sep 2014 11:02:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1248) TCP gaps inserted in wrong place in
 HTTP range request
In-Reply-To: <JIRA.15906.1410364921000@Atlassian.JIRA>
References: <JIRA.15906.1410364921000@Atlassian.JIRA>
	<JIRA.15906.1410364921116@bro-tracker.atlassian.net>
Message-ID: <JIRA.15906.1410364921000.36.1410364927621@Atlassian.JIRA>

Jimmy Jones created BIT-1248:
--------------------------------

             Summary: TCP gaps inserted in wrong place in HTTP range request
                 Key: BIT-1248
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1248
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: git/master
         Environment: CentOS 6
            Reporter: Jimmy Jones
         Attachments: http-range-hole1.pcap, http-range.pcap

See attached testcases, one with packet #10 missing.

Putting this through the file extraction framework with the script below, the hole is not inserted at the correct point (the data either side of the hole is side by side). I believe this may be because HTTP.cc calls DataIn with an offset argument, which isn't updated for missing packets.

Bug still exists with BIT-1240 applied.

event file_new(f: fa_file)
{ Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]); } 



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Wed Sep 10 10:04:07 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Wed, 10 Sep 2014 12:04:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1249) Trouble with pcap interfaces with
 colons on their name
In-Reply-To: <JIRA.15907.1410368617000@Atlassian.JIRA>
References: <JIRA.15907.1410368617000@Atlassian.JIRA>
	<JIRA.15907.1410368617282@bro-tracker.atlassian.net>
Message-ID: <JIRA.15907.1410368617000.37.1410368647656@Atlassian.JIRA>

Robin Sommer created BIT-1249:
---------------------------------

             Summary: Trouble with pcap interfaces with colons on their name
                 Key: BIT-1249
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1249
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: git/master
            Reporter: Robin Sommer
            Assignee: Robin Sommer


Pcap interfaces can have colons in their name, which confuses the new packet source code, as it uses "xxx:" to identify the type of source to use (e.g., "dag0:0" isn't handed over to pcap; instead Bro looks for a plugin "dag0" to handle the interface "0".)

As a work-around, it should work to use "-i pcap:dag:0".

Possible fixes:
  - we use a different prefix separator (e.g., "-i netmap!em3")
  - fallback to pcap if prefix not known.
  - make the work around the solution (i.e., if in doubt, specify pcap explicitly)

Reported by John Donaldson.




--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Wed Sep 10 10:04:08 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Wed, 10 Sep 2014 12:04:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1249) Trouble with pcap interfaces with
 colons in their name
In-Reply-To: <JIRA.15907.1410368617000@Atlassian.JIRA>
References: <JIRA.15907.1410368617000@Atlassian.JIRA>
	<JIRA.15907.1410368617282@bro-tracker.atlassian.net>
Message-ID: <JIRA.15907.1410368617000.38.1410368648228@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robin Sommer updated BIT-1249:
------------------------------
    Summary: Trouble with pcap interfaces with colons in their name  (was: Trouble with pcap interfaces with colons on their name)

> Trouble with pcap interfaces with colons in their name
> ------------------------------------------------------
>
>                 Key: BIT-1249
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1249
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>            Reporter: Robin Sommer
>            Assignee: Robin Sommer
>
> Pcap interfaces can have colons in their name, which confuses the new packet source code, as it uses "xxx:" to identify the type of source to use (e.g., "dag0:0" isn't handed over to pcap; instead Bro looks for a plugin "dag0" to handle the interface "0".)
> As a work-around, it should work to use "-i pcap:dag:0".
> Possible fixes:
>   - we use a different prefix separator (e.g., "-i netmap!em3")
>   - fallback to pcap if prefix not known.
>   - make the work around the solution (i.e., if in doubt, specify pcap explicitly)
> Reported by John Donaldson.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Wed Sep 10 11:32:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Wed, 10 Sep 2014 13:32:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1240) TCP gaps inserted in wrong place
In-Reply-To: <JIRA.15802.1410036351000@Atlassian.JIRA>
References: <JIRA.15802.1410036351000@Atlassian.JIRA>
	<JIRA.15802.1410036351678@bro-tracker.atlassian.net>
Message-ID: <JIRA.15802.1410036351000.40.1410373927712@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18008#comment-18008 ] 

Jon Siwek commented on BIT-1240:
--------------------------------

You're right, http_entity_data was still raised in an odd order.  There's a new commit on the same branch, topic/jsiwek/bit-1240.  If you're able, please pull that in and let me know if it helps.

> TCP gaps inserted in wrong place
> --------------------------------
>
>                 Key: BIT-1240
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1240
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>             Fix For: 2.4
>
>         Attachments: get-hole1.trace
>
>
> Using attached test file, I tried using the file analysis framework to extract out the payload, which is a copy of one of the bro testcases but with a packet removed.
> However the extracted file has nulls padded at the beginning, but they should be at offset 1448. Looking at what happens, the gap is signalled to File::Gap before the data is received in File::DataIn, which calculates the offset to write the payload by adding the seen bytes to the missing byte count. Should gaps always be signalled in order with the data - this also affects users of content_gap, who would receive the data and hole "out of order"?
> Used the following bro script:
> event file_new(f: fa_file)
> {
>   Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]);
> }



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From noreply at bro.org  Thu Sep 11 00:00:39 2014
From: noreply at bro.org (Merge Tracker)
Date: Thu, 11 Sep 2014 00:00:39 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409110700.s8B70dcM007351@bro-ids.icir.org>


Open GitHub Pull Requests
=========================

Issue    Component    User        Updated     Title
-------  -----------  ----------  ----------  ------------------------
#1 [1]   bro-scripts  cmavr8 [2]  2014-09-09  Fixed typo in README [3]


[1]  Pull Request #1             https://github.com/bro/bro-scripts/pull/1
[2]  cmavr8                      https://github.com/cmavr8
[3]  Merge Pull Request #1 with  git pull --no-ff --no-commit https://github.com/cmavr8/bro-scripts.git master


From jira at bro-tracker.atlassian.net  Thu Sep 11 02:10:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Thu, 11 Sep 2014 04:10:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1240) TCP gaps inserted in wrong place
In-Reply-To: <JIRA.15802.1410036351000@Atlassian.JIRA>
References: <JIRA.15802.1410036351000@Atlassian.JIRA>
	<JIRA.15802.1410036351678@bro-tracker.atlassian.net>
Message-ID: <JIRA.15802.1410036351000.41.1410426607664@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18009#comment-18009 ] 

Jimmy Jones commented on BIT-1240:
----------------------------------

Just run through my samples and works as expected. Thanks Jon.

> TCP gaps inserted in wrong place
> --------------------------------
>
>                 Key: BIT-1240
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1240
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>             Fix For: 2.4
>
>         Attachments: get-hole1.trace
>
>
> Using attached test file, I tried using the file analysis framework to extract out the payload, which is a copy of one of the bro testcases but with a packet removed.
> However the extracted file has nulls padded at the beginning, but they should be at offset 1448. Looking at what happens, the gap is signalled to File::Gap before the data is received in File::DataIn, which calculates the offset to write the payload by adding the seen bytes to the missing byte count. Should gaps always be signalled in order with the data - this also affects users of content_gap, who would receive the data and hole "out of order"?
> Used the following bro script:
> event file_new(f: fa_file)
> {
>   Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]);
> }



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Thu Sep 11 08:51:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Thu, 11 Sep 2014 10:51:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1246) Missing last TCP segment
In-Reply-To: <JIRA.15904.1410361694000@Atlassian.JIRA>
References: <JIRA.15904.1410361694000@Atlassian.JIRA>
	<JIRA.15904.1410361694392@bro-tracker.atlassian.net>
Message-ID: <JIRA.15904.1410361694000.43.1410450667789@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1246:
---------------------------
    Fix Version/s: 2.4

> Missing last TCP segment
> ------------------------
>
>                 Key: BIT-1246
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1246
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>             Fix For: 2.4
>
>         Attachments: get-hole2.trace
>
>
> Possibly related to BIT-1240, I've got a sample (attached) where the penultimate TCP segment of the http response is missing.
> Using the file analysis framework I'm extracting the payload, but I don't get any of the last segment. Without the fix in BIT-1240 additionally the NULL padding for the missing packets happens in the wrong place.
> Used the following bro script:
> event file_new(f: fa_file)
> { Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]); } 



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Thu Sep 11 08:55:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Thu, 11 Sep 2014 10:55:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1246) Missing last TCP segment
In-Reply-To: <JIRA.15904.1410361694000@Atlassian.JIRA>
References: <JIRA.15904.1410361694000@Atlassian.JIRA>
	<JIRA.15904.1410361694392@bro-tracker.atlassian.net>
Message-ID: <JIRA.15904.1410361694000.45.1410450907761@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18010#comment-18010 ] 

Jon Siwek commented on BIT-1246:
--------------------------------

This looked like an issue with Bro's TCP reassembly.  Branch "topic/jsiwek/bit-1246" has a fix if you want to test.  (That branch also includes the changes from BIT-1240, but those aren't strictly related/dependent, they just made testing/verification easier).

> Missing last TCP segment
> ------------------------
>
>                 Key: BIT-1246
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1246
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>             Fix For: 2.4
>
>         Attachments: get-hole2.trace
>
>
> Possibly related to BIT-1240, I've got a sample (attached) where the penultimate TCP segment of the http response is missing.
> Using the file analysis framework I'm extracting the payload, but I don't get any of the last segment. Without the fix in BIT-1240 additionally the NULL padding for the missing packets happens in the wrong place.
> Used the following bro script:
> event file_new(f: fa_file)
> { Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]); } 



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Thu Sep 11 10:31:08 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Thu, 11 Sep 2014 12:31:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1248) TCP gaps inserted in wrong place in
 HTTP range request
In-Reply-To: <JIRA.15906.1410364921000@Atlassian.JIRA>
References: <JIRA.15906.1410364921000@Atlassian.JIRA>
	<JIRA.15906.1410364921116@bro-tracker.atlassian.net>
Message-ID: <JIRA.15906.1410364921000.47.1410456668327@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18011#comment-18011 ] 

Jon Siwek commented on BIT-1248:
--------------------------------

Yeah, looked like the problem here was what you were thinking.  Branch "topic/jsiwek/bit-1248" has a fix in it (includes changes from BIT-1240 and BIT-1246).

> TCP gaps inserted in wrong place in HTTP range request
> ------------------------------------------------------
>
>                 Key: BIT-1248
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1248
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: http-range-hole1.pcap, http-range.pcap
>
>
> See attached testcases, one with packet #10 missing.
> Putting this through the file extraction framework with the script below, the hole is not inserted at the correct point (the data either side of the hole is side by side). I believe this may be because HTTP.cc calls DataIn with an offset argument, which isn't updated for missing packets.
> Bug still exists with BIT-1240 applied.
> event file_new(f: fa_file)
> { Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]); } 



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Thu Sep 11 13:13:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Thu, 11 Sep 2014 15:13:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1247) Missing packet in HTTP byte ranges
 request stops processing
In-Reply-To: <JIRA.15905.1410363677000@Atlassian.JIRA>
References: <JIRA.15905.1410363677000@Atlassian.JIRA>
	<JIRA.15905.1410363677334@bro-tracker.atlassian.net>
Message-ID: <JIRA.15905.1410363677000.49.1410466387715@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18012#comment-18012 ] 

Jon Siwek commented on BIT-1247:
--------------------------------

The logic to check whether the gap fell entirely within entity body was off.  A fix is in "topic/jsiwek/bit-1247".  That also includes changes from BIT-1240, BIT-1246, and BIT-1248.  I also have "topic/jsiwek/jj-bugs" that includes everything; if I need to do more changes I may just do them there otherwise may be hard to keep track of things, else we'll see about getting this merged in to master if it looks good to you.

> Missing packet in HTTP byte ranges request stops processing
> -----------------------------------------------------------
>
>                 Key: BIT-1247
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1247
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: byteranges-hole1.trace
>
>
> I've created the attached file from one in the testing framework, but with packet #8 removed. The missing packet is in the middle of a mime part and doesn't straddle any MIME boundaries. However with the packet removed, the file output by the file analysis framework only contains the data up until the missing packet. As the missing packet didn't include any MIME boundaries, I wouldn't expect this behavior.
> Used the following bro script:
> event file_new(f: fa_file)
> { Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]); } 



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Thu Sep 11 13:47:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Thu, 11 Sep 2014 15:47:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1246) Missing last TCP segment
In-Reply-To: <JIRA.15904.1410361694000@Atlassian.JIRA>
References: <JIRA.15904.1410361694000@Atlassian.JIRA>
	<JIRA.15904.1410361694392@bro-tracker.atlassian.net>
Message-ID: <JIRA.15904.1410361694000.50.1410468427657@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18013#comment-18013 ] 

Jimmy Jones commented on BIT-1246:
----------------------------------

Thanks Jon, tests work for me as expected now.

> Missing last TCP segment
> ------------------------
>
>                 Key: BIT-1246
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1246
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>             Fix For: 2.4
>
>         Attachments: get-hole2.trace
>
>
> Possibly related to BIT-1240, I've got a sample (attached) where the penultimate TCP segment of the http response is missing.
> Using the file analysis framework I'm extracting the payload, but I don't get any of the last segment. Without the fix in BIT-1240 additionally the NULL padding for the missing packets happens in the wrong place.
> Used the following bro script:
> event file_new(f: fa_file)
> { Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]); } 



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Thu Sep 11 13:47:08 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Thu, 11 Sep 2014 15:47:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1248) TCP gaps inserted in wrong place in
 HTTP range request
In-Reply-To: <JIRA.15906.1410364921000@Atlassian.JIRA>
References: <JIRA.15906.1410364921000@Atlassian.JIRA>
	<JIRA.15906.1410364921116@bro-tracker.atlassian.net>
Message-ID: <JIRA.15906.1410364921000.51.1410468428214@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18014#comment-18014 ] 

Jimmy Jones commented on BIT-1248:
----------------------------------

Cool, works for me now.

> TCP gaps inserted in wrong place in HTTP range request
> ------------------------------------------------------
>
>                 Key: BIT-1248
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1248
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: http-range-hole1.pcap, http-range.pcap
>
>
> See attached testcases, one with packet #10 missing.
> Putting this through the file extraction framework with the script below, the hole is not inserted at the correct point (the data either side of the hole is side by side). I believe this may be because HTTP.cc calls DataIn with an offset argument, which isn't updated for missing packets.
> Bug still exists with BIT-1240 applied.
> event file_new(f: fa_file)
> { Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]); } 



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Thu Sep 11 13:49:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Thu, 11 Sep 2014 15:49:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1247) Missing packet in HTTP byte ranges
 request stops processing
In-Reply-To: <JIRA.15905.1410363677000@Atlassian.JIRA>
References: <JIRA.15905.1410363677000@Atlassian.JIRA>
	<JIRA.15905.1410363677334@bro-tracker.atlassian.net>
Message-ID: <JIRA.15905.1410363677000.52.1410468547622@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18015#comment-18015 ] 

Jimmy Jones commented on BIT-1247:
----------------------------------

Tested and looks good! Will try and do some more testing next week and raise anything I find.

> Missing packet in HTTP byte ranges request stops processing
> -----------------------------------------------------------
>
>                 Key: BIT-1247
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1247
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: byteranges-hole1.trace
>
>
> I've created the attached file from one in the testing framework, but with packet #8 removed. The missing packet is in the middle of a mime part and doesn't straddle any MIME boundaries. However with the packet removed, the file output by the file analysis framework only contains the data up until the missing packet. As the missing packet didn't include any MIME boundaries, I wouldn't expect this behavior.
> Used the following bro script:
> event file_new(f: fa_file)
> { Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]); } 



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Thu Sep 11 14:33:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Thu, 11 Sep 2014 16:33:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1247) Missing packet in HTTP byte ranges
 request stops processing
In-Reply-To: <JIRA.15905.1410363677000@Atlassian.JIRA>
References: <JIRA.15905.1410363677000@Atlassian.JIRA>
	<JIRA.15905.1410363677334@bro-tracker.atlassian.net>
Message-ID: <JIRA.15905.1410363677000.56.1410471187821@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1247:
---------------------------
    Fix Version/s: 2.4

> Missing packet in HTTP byte ranges request stops processing
> -----------------------------------------------------------
>
>                 Key: BIT-1247
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1247
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>             Fix For: 2.4
>
>         Attachments: byteranges-hole1.trace
>
>
> I've created the attached file from one in the testing framework, but with packet #8 removed. The missing packet is in the middle of a mime part and doesn't straddle any MIME boundaries. However with the packet removed, the file output by the file analysis framework only contains the data up until the missing packet. As the missing packet didn't include any MIME boundaries, I wouldn't expect this behavior.
> Used the following bro script:
> event file_new(f: fa_file)
> { Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]); } 



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Thu Sep 11 14:33:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Thu, 11 Sep 2014 16:33:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1248) TCP gaps inserted in wrong place in
 HTTP range request
In-Reply-To: <JIRA.15906.1410364921000@Atlassian.JIRA>
References: <JIRA.15906.1410364921000@Atlassian.JIRA>
	<JIRA.15906.1410364921116@bro-tracker.atlassian.net>
Message-ID: <JIRA.15906.1410364921000.54.1410471187725@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1248:
---------------------------
    Fix Version/s: 2.4

> TCP gaps inserted in wrong place in HTTP range request
> ------------------------------------------------------
>
>                 Key: BIT-1248
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1248
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>             Fix For: 2.4
>
>         Attachments: http-range-hole1.pcap, http-range.pcap
>
>
> See attached testcases, one with packet #10 missing.
> Putting this through the file extraction framework with the script below, the hole is not inserted at the correct point (the data either side of the hole is side by side). I believe this may be because HTTP.cc calls DataIn with an offset argument, which isn't updated for missing packets.
> Bug still exists with BIT-1240 applied.
> event file_new(f: fa_file)
> { Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]); } 



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Thu Sep 11 14:38:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Thu, 11 Sep 2014 16:38:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1250) topic/jsiwek/jj-bugs
In-Reply-To: <JIRA.15908.1410471445000@Atlassian.JIRA>
References: <JIRA.15908.1410471445000@Atlassian.JIRA>
	<JIRA.15908.1410471445032@bro-tracker.atlassian.net>
Message-ID: <JIRA.15908.1410471445000.57.1410471487631@Atlassian.JIRA>

Jon Siwek created BIT-1250:
------------------------------

             Summary: topic/jsiwek/jj-bugs
                 Key: BIT-1250
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1250
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: git/master
            Reporter: Jon Siwek
             Fix For: 2.4


This branch is in bro, bro-testing, and bro-testing-private repos, fixing problems reported in BIT-1240, BIT-1246, BIT-1247, and BIT-1248.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Thu Sep 11 14:38:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Thu, 11 Sep 2014 16:38:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1250) topic/jsiwek/jj-bugs
In-Reply-To: <JIRA.15908.1410471445000@Atlassian.JIRA>
References: <JIRA.15908.1410471445000@Atlassian.JIRA>
	<JIRA.15908.1410471445032@bro-tracker.atlassian.net>
Message-ID: <JIRA.15908.1410471445000.58.1410471487687@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1250:
---------------------------
    Status: Merge Request  (was: Open)

> topic/jsiwek/jj-bugs
> --------------------
>
>                 Key: BIT-1250
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1250
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>            Reporter: Jon Siwek
>             Fix For: 2.4
>
>
> This branch is in bro, bro-testing, and bro-testing-private repos, fixing problems reported in BIT-1240, BIT-1246, BIT-1247, and BIT-1248.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From noreply at bro.org  Fri Sep 12 00:00:27 2014
From: noreply at bro.org (Merge Tracker)
Date: Fri, 12 Sep 2014 00:00:27 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409120700.s8C70RdW017203@bro-ids.icir.org>


Open Merge Requests
===================

ID            Component    Reporter    Assignee    Updated       For Version  Priority    Summary
------------  -----------  ----------  ----------  ----------  -------------  ----------  ------------------------
BIT-1250 [1]  Bro          Jon Siwek   -           2014-09-11            2.4  Normal      topic/jsiwek/jj-bugs [2]


[1]  BIT-1250  https://bro-tracker.atlassian.net/browse/BIT-1250
[2]  jj-bugs   https://github.com/bro/bro/tree/topic/jsiwek/jj-bugs


From seth at icir.org  Fri Sep 12 06:10:12 2014
From: seth at icir.org (Seth Hall)
Date: Fri, 12 Sep 2014 09:10:12 -0400
Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/jsiwek/bit-1246: Fix
	issue w/ TCP reassembler not delivering some segments. (f1cef9d)
In-Reply-To: <201409111559.s8BFxcBU013288@bro-ids.icir.org>
References: <201409111559.s8BFxcBU013288@bro-ids.icir.org>
Message-ID: <3266A423-4258-4E8E-8B57-4C6F747C4AE6@icir.org>


On Sep 11, 2014, at 11:59 AM, Jonathan Siwek <jsiwek at ncsa.illinois.edu> wrote:

> +	// Only report on content gaps for connections that
> +	// are in a cleanly established state.  In other
> +	// states, these can arise falsely due to things
> +	// like sequence number mismatches in RSTs, or
> +	// unseen previous packets in partial connections.
> +	// The one opportunity we lose here is on clean FIN
> +	// handshakes, but Oh Well.

If I'm reading this right, this seems like an undesirable outcome.  If Bro starts and a connection is in the middle, does this mean we wouldn't see any content gaps for that connection?

.Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From jsiwek at illinois.edu  Fri Sep 12 06:44:34 2014
From: jsiwek at illinois.edu (Siwek, Jon)
Date: Fri, 12 Sep 2014 13:44:34 +0000
Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/jsiwek/bit-1246: Fix
 issue w/ TCP reassembler not delivering some segments. (f1cef9d)
In-Reply-To: <3266A423-4258-4E8E-8B57-4C6F747C4AE6@icir.org>
References: <201409111559.s8BFxcBU013288@bro-ids.icir.org>
	<3266A423-4258-4E8E-8B57-4C6F747C4AE6@icir.org>
Message-ID: <A250F72F-20EC-4282-9591-DD75E0474860@illinois.edu>


> On Sep 12, 2014, at 8:10 AM, Seth Hall <seth at icir.org> wrote:
> 
> On Sep 11, 2014, at 11:59 AM, Jonathan Siwek <jsiwek at ncsa.illinois.edu> wrote:
> 
>> +	// Only report on content gaps for connections that
>> +	// are in a cleanly established state.  In other
>> +	// states, these can arise falsely due to things
>> +	// like sequence number mismatches in RSTs, or
>> +	// unseen previous packets in partial connections.
>> +	// The one opportunity we lose here is on clean FIN
>> +	// handshakes, but Oh Well.
> 
> If I'm reading this right, this seems like an undesirable outcome.  If Bro starts and a connection is in the middle, does this mean we wouldn't see any content gaps for that connection?

Yes, I think that may be the case, but just for the content_gap event, not for telling analyzers there?s a gap in the stream.  It?s adjustable by redef'ing BifConst::report_gaps_for_partial.  It?s also not new behavior, that comment was attached to some already-existing code that I factored out in to a separate function so I could easily re-use it.  Not giving judgement on what behavior should be the default, but changing it shouldn?t be done as part of what I was trying to fix in this commit.

- Jon


From seth at icir.org  Fri Sep 12 06:58:22 2014
From: seth at icir.org (Seth Hall)
Date: Fri, 12 Sep 2014 09:58:22 -0400
Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/jsiwek/bit-1246: Fix
	issue w/ TCP reassembler not delivering some segments. (f1cef9d)
In-Reply-To: <A250F72F-20EC-4282-9591-DD75E0474860@illinois.edu>
References: <201409111559.s8BFxcBU013288@bro-ids.icir.org>
	<3266A423-4258-4E8E-8B57-4C6F747C4AE6@icir.org>
	<A250F72F-20EC-4282-9591-DD75E0474860@illinois.edu>
Message-ID: <FCA86D97-76CD-4D55-B385-8CFF2EB46B5E@icir.org>


On Sep 12, 2014, at 9:44 AM, Siwek, Jon <jsiwek at illinois.edu> wrote:

> Yes, I think that may be the case, but just for the content_gap event, not for telling analyzers there?s a gap in the stream.  It?s adjustable by redef'ing BifConst::report_gaps_for_partial.  It?s also not new behavior, that comment was attached to some already-existing code that I factored out in to a separate function so I could easily re-use it.  Not giving judgement on what behavior should be the default, but changing it shouldn?t be done as part of what I was trying to fix in this commit.

Ah!  I remember this now.  Thanks for explaining.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From jira at bro-tracker.atlassian.net  Fri Sep 12 16:23:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Fri, 12 Sep 2014 18:23:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1251) content_gap not being raised
In-Reply-To: <JIRA.15909.1410564156000@Atlassian.JIRA>
References: <JIRA.15909.1410564156000@Atlassian.JIRA>
	<JIRA.15909.1410564156686@bro-tracker.atlassian.net>
Message-ID: <JIRA.15909.1410564156000.59.1410564187645@Atlassian.JIRA>

Jimmy Jones created BIT-1251:
--------------------------------

             Summary: content_gap not being raised
                 Key: BIT-1251
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1251
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: git/master
         Environment: Fedora 20
            Reporter: Jimmy Jones
         Attachments: analyser.bro, test5-10mb.pcap.gz

Using the attached bro script, I extract out the http response, which contains some null padding for the missing packets. However the content_gap event never gets raised, despite http_entity_data not receiving the entire download.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Fri Sep 12 19:48:07 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Fri, 12 Sep 2014 21:48:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1250) topic/jsiwek/jj-bugs
In-Reply-To: <JIRA.15908.1410471445000@Atlassian.JIRA>
References: <JIRA.15908.1410471445000@Atlassian.JIRA>
	<JIRA.15908.1410471445032@bro-tracker.atlassian.net>
Message-ID: <JIRA.15908.1410471445000.61.1410576487725@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18016#comment-18016 ] 

Robin Sommer commented on BIT-1250:
-----------------------------------

Merged, but two questions:

    - HTTP_Message::entity_data_buffer: that's allocated with every
      HTTP message. Would there be any benefit to allocate it only on
      demand when it's actually used? I can't immediately tell if the
      buffer is indeed going to be accessed generally with every message, or
      only sometimes.

    - I'm seeing somewhat longer execution times:

      {noformat}
      [ 28%] tests.m57-short ... failed (+1.0%)
      [ 50%] tests.short ... failed (+1.5%)
      [ 75%] tests.medium ... failed (+1.4%)
      {noformat}

      Could this even be related to that buffer above?

      Oddly enough, I'm seeing *better* times for the v6 trace:

      {noformat}
      [ 71%] tests.ipv6 ... failed (-1.5%)
      {noformat}

      (These failures are consistent across multiple runs; sometimes I get
      false alarms with the timing measurements, but normally they
      disappear on reruns.)

Closing the individual tickets but leaving this ticket open for now.
Please check and close if everything looks good / you can't reproduce
the timing differences.

> topic/jsiwek/jj-bugs
> --------------------
>
>                 Key: BIT-1250
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1250
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>            Reporter: Jon Siwek
>             Fix For: 2.4
>
>
> This branch is in bro, bro-testing, and bro-testing-private repos, fixing problems reported in BIT-1240, BIT-1246, BIT-1247, and BIT-1248.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From jira at bro-tracker.atlassian.net  Fri Sep 12 19:56:08 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Fri, 12 Sep 2014 21:56:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1250) topic/jsiwek/jj-bugs
In-Reply-To: <JIRA.15908.1410471445000@Atlassian.JIRA>
References: <JIRA.15908.1410471445000@Atlassian.JIRA>
	<JIRA.15908.1410471445032@bro-tracker.atlassian.net>
Message-ID: <JIRA.15908.1410471445000.63.1410576968166@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robin Sommer updated BIT-1250:
------------------------------
    Status: Open  (was: Merge Request)

> topic/jsiwek/jj-bugs
> --------------------
>
>                 Key: BIT-1250
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1250
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>            Reporter: Jon Siwek
>             Fix For: 2.4
>
>
> This branch is in bro, bro-testing, and bro-testing-private repos, fixing problems reported in BIT-1240, BIT-1246, BIT-1247, and BIT-1248.



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)

From noreply at bro.org  Sun Sep 14 00:00:20 2014
From: noreply at bro.org (Merge Tracker)
Date: Sun, 14 Sep 2014 00:00:20 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409140700.s8E70KgL030736@bro-ids.icir.org>


Open GitHub Pull Requests
=========================

Issue    Component      User             Updated     Title
-------  -------------  ---------------  ----------  ----------------------
#2 [1]   packet-bricks  asifjamshed [2]  2014-09-13  typo [3]
#1 [4]   packet-bricks  asifjamshed [5]  2014-09-13  Fixing minor typos [6]


[1]  Pull Request #2             https://github.com/bro/packet-bricks/pull/2
[2]  asifjamshed                 https://github.com/asifjamshed
[3]  Merge Pull Request #2 with  git pull --no-ff --no-commit https://github.com/asifjamshed/packet-bricks.git master
[4]  Pull Request #1             https://github.com/bro/packet-bricks/pull/1
[5]  asifjamshed                 https://github.com/asifjamshed
[6]  Merge Pull Request #1 with  git pull --no-ff --no-commit https://github.com/asifjamshed/packet-bricks.git patch-1


From noreply at bro.org  Mon Sep 15 00:00:33 2014
From: noreply at bro.org (Merge Tracker)
Date: Mon, 15 Sep 2014 00:00:33 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409150700.s8F70Xox004682@bro-ids.icir.org>


Open GitHub Pull Requests
=========================

Issue    Component      User             Updated     Title
-------  -------------  ---------------  ----------  ----------------------
#2 [1]   packet-bricks  asifjamshed [2]  2014-09-13  typo [3]
#1 [4]   packet-bricks  asifjamshed [5]  2014-09-13  Fixing minor typos [6]


[1]  Pull Request #2             https://github.com/bro/packet-bricks/pull/2
[2]  asifjamshed                 https://github.com/asifjamshed
[3]  Merge Pull Request #2 with  git pull --no-ff --no-commit https://github.com/asifjamshed/packet-bricks.git master
[4]  Pull Request #1             https://github.com/bro/packet-bricks/pull/1
[5]  asifjamshed                 https://github.com/asifjamshed
[6]  Merge Pull Request #1 with  git pull --no-ff --no-commit https://github.com/asifjamshed/packet-bricks.git patch-1


From jira at bro-tracker.atlassian.net  Mon Sep 15 09:47:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Mon, 15 Sep 2014 11:47:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1246) Missing last TCP segment
In-Reply-To: <JIRA.15904.1410361694000@Atlassian.JIRA>
References: <JIRA.15904.1410361694000@Atlassian.JIRA>
	<JIRA.15904.1410361694392@bro-tracker.atlassian.net>
Message-ID: <JIRA.15904.1410361694000.4.1410799627950@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1246:
---------------------------
    Resolution: Fixed
        Status: Closed  (was: Open)

> Missing last TCP segment
> ------------------------
>
>                 Key: BIT-1246
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1246
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>             Fix For: 2.4
>
>         Attachments: get-hole2.trace
>
>
> Possibly related to BIT-1240, I've got a sample (attached) where the penultimate TCP segment of the http response is missing.
> Using the file analysis framework I'm extracting the payload, but I don't get any of the last segment. Without the fix in BIT-1240 additionally the NULL padding for the missing packets happens in the wrong place.
> Used the following bro script:
> event file_new(f: fa_file)
> { Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]); } 



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 15 09:47:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Mon, 15 Sep 2014 11:47:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1240) TCP gaps inserted in wrong place
In-Reply-To: <JIRA.15802.1410036351000@Atlassian.JIRA>
References: <JIRA.15802.1410036351000@Atlassian.JIRA>
	<JIRA.15802.1410036351678@bro-tracker.atlassian.net>
Message-ID: <JIRA.15802.1410036351000.2.1410799627851@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1240:
---------------------------
    Resolution: Fixed
        Status: Closed  (was: Open)

> TCP gaps inserted in wrong place
> --------------------------------
>
>                 Key: BIT-1240
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1240
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>             Fix For: 2.4
>
>         Attachments: get-hole1.trace
>
>
> Using attached test file, I tried using the file analysis framework to extract out the payload, which is a copy of one of the bro testcases but with a packet removed.
> However the extracted file has nulls padded at the beginning, but they should be at offset 1448. Looking at what happens, the gap is signalled to File::Gap before the data is received in File::DataIn, which calculates the offset to write the payload by adding the seen bytes to the missing byte count. Should gaps always be signalled in order with the data - this also affects users of content_gap, who would receive the data and hole "out of order"?
> Used the following bro script:
> event file_new(f: fa_file)
> {
>   Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]);
> }



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 15 09:48:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Mon, 15 Sep 2014 11:48:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1247) Missing packet in HTTP byte ranges
 request stops processing
In-Reply-To: <JIRA.15905.1410363677000@Atlassian.JIRA>
References: <JIRA.15905.1410363677000@Atlassian.JIRA>
	<JIRA.15905.1410363677334@bro-tracker.atlassian.net>
Message-ID: <JIRA.15905.1410363677000.6.1410799687655@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1247:
---------------------------
    Resolution: Fixed
        Status: Closed  (was: Open)

> Missing packet in HTTP byte ranges request stops processing
> -----------------------------------------------------------
>
>                 Key: BIT-1247
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1247
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>             Fix For: 2.4
>
>         Attachments: byteranges-hole1.trace
>
>
> I've created the attached file from one in the testing framework, but with packet #8 removed. The missing packet is in the middle of a mime part and doesn't straddle any MIME boundaries. However with the packet removed, the file output by the file analysis framework only contains the data up until the missing packet. As the missing packet didn't include any MIME boundaries, I wouldn't expect this behavior.
> Used the following bro script:
> event file_new(f: fa_file)
> { Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]); } 



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 15 09:48:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Mon, 15 Sep 2014 11:48:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1248) TCP gaps inserted in wrong place in
 HTTP range request
In-Reply-To: <JIRA.15906.1410364921000@Atlassian.JIRA>
References: <JIRA.15906.1410364921000@Atlassian.JIRA>
	<JIRA.15906.1410364921116@bro-tracker.atlassian.net>
Message-ID: <JIRA.15906.1410364921000.7.1410799687706@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1248:
---------------------------
    Resolution: Fixed
        Status: Closed  (was: Open)

> TCP gaps inserted in wrong place in HTTP range request
> ------------------------------------------------------
>
>                 Key: BIT-1248
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1248
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>             Fix For: 2.4
>
>         Attachments: http-range-hole1.pcap, http-range.pcap
>
>
> See attached testcases, one with packet #10 missing.
> Putting this through the file extraction framework with the script below, the hole is not inserted at the correct point (the data either side of the hole is side by side). I believe this may be because HTTP.cc calls DataIn with an offset argument, which isn't updated for missing packets.
> Bug still exists with BIT-1240 applied.
> event file_new(f: fa_file)
> { Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]); } 



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 15 10:45:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Mon, 15 Sep 2014 12:45:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1251) content_gap not being raised
In-Reply-To: <JIRA.15909.1410564156000@Atlassian.JIRA>
References: <JIRA.15909.1410564156000@Atlassian.JIRA>
	<JIRA.15909.1410564156686@bro-tracker.atlassian.net>
Message-ID: <JIRA.15909.1410564156000.10.1410803107726@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18100#comment-18100 ] 

Jon Siwek commented on BIT-1251:
--------------------------------

"content_gap" is intentionally (or maybe just historically... hard to tell) not raised in some situations.  Particularly if Bro doesn't consider a connection fully established, it may not raise it unless you "redef report_gaps_for_partial=T;".  In this case, the default behavior has a detrimental interaction with TCP reassembly -- we miss a segment from the server and start buffering further segments (because we don't know for sure if it will come out of order later), the server sends a FIN and so Bro considers the server endpoint closed, finally the client ACKs enough that we can tell there's a gap, but we don't raise "content_gap" because the connection is no longer "fully established".

Addressing some root problems here might take significant changes/rewrite of the TCP analysis, so you might check in to the workaround of redef'ing "report_gaps_for_partial" (source code comments claim a downside of that is raising "content_gap" falsely in more situations, but not sure how common/relevant that still is).

> content_gap not being raised
> ----------------------------
>
>                 Key: BIT-1251
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1251
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>         Environment: Fedora 20
>            Reporter: Jimmy Jones
>         Attachments: analyser.bro, test5-10mb.pcap.gz
>
>
> Using the attached bro script, I extract out the http response, which contains some null padding for the missing packets. However the content_gap event never gets raised, despite http_entity_data not receiving the entire download.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 15 11:43:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Mon, 15 Sep 2014 13:43:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1250) topic/jsiwek/jj-bugs
In-Reply-To: <JIRA.15908.1410471445000@Atlassian.JIRA>
References: <JIRA.15908.1410471445000@Atlassian.JIRA>
	<JIRA.15908.1410471445032@bro-tracker.atlassian.net>
Message-ID: <JIRA.15908.1410471445000.11.1410806587653@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18101#comment-18101 ] 

Jon Siwek commented on BIT-1250:
--------------------------------

{quote}
HTTP_Message::entity_data_buffer: that's allocated with every
HTTP message. Would there be any benefit to allocate it only on
demand when it's actually used? I can't immediately tell if the
buffer is indeed going to be accessed generally with every message, or
only sometimes.
{quote}

Good point. It is only used sometimes.

{quote}
I'm seeing somewhat longer execution times:
Could this even be related to that buffer above?
{quote}

Doing some quick timings of test.medium I got a similar result: +1.6%.  Changing that buffer to be allocated on-demand seemed to make the difference negligible, so I think that explains it.  And IIRC, the original code also did allocation on-demand.  I'll commit the change to master.

> topic/jsiwek/jj-bugs
> --------------------
>
>                 Key: BIT-1250
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1250
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>            Reporter: Jon Siwek
>             Fix For: 2.4
>
>
> This branch is in bro, bro-testing, and bro-testing-private repos, fixing problems reported in BIT-1240, BIT-1246, BIT-1247, and BIT-1248.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 15 11:44:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Mon, 15 Sep 2014 13:44:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1250) topic/jsiwek/jj-bugs
In-Reply-To: <JIRA.15908.1410471445000@Atlassian.JIRA>
References: <JIRA.15908.1410471445000@Atlassian.JIRA>
	<JIRA.15908.1410471445032@bro-tracker.atlassian.net>
Message-ID: <JIRA.15908.1410471445000.12.1410806647627@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1250:
---------------------------
    Resolution: Merged
        Status: Closed  (was: Open)

> topic/jsiwek/jj-bugs
> --------------------
>
>                 Key: BIT-1250
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1250
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>            Reporter: Jon Siwek
>             Fix For: 2.4
>
>
> This branch is in bro, bro-testing, and bro-testing-private repos, fixing problems reported in BIT-1240, BIT-1246, BIT-1247, and BIT-1248.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From noreply at bro.org  Tue Sep 16 00:00:19 2014
From: noreply at bro.org (Merge Tracker)
Date: Tue, 16 Sep 2014 00:00:19 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409160700.s8G70Jqt005681@bro-ids.icir.org>


Open GitHub Pull Requests
=========================

Issue    Component      User             Updated     Title
-------  -------------  ---------------  ----------  ----------------------
#2 [1]   packet-bricks  asifjamshed [2]  2014-09-13  typo [3]
#1 [4]   packet-bricks  asifjamshed [5]  2014-09-13  Fixing minor typos [6]


[1]  Pull Request #2             https://github.com/bro/packet-bricks/pull/2
[2]  asifjamshed                 https://github.com/asifjamshed
[3]  Merge Pull Request #2 with  git pull --no-ff --no-commit https://github.com/asifjamshed/packet-bricks.git master
[4]  Pull Request #1             https://github.com/bro/packet-bricks/pull/1
[5]  asifjamshed                 https://github.com/asifjamshed
[6]  Merge Pull Request #1 with  git pull --no-ff --no-commit https://github.com/asifjamshed/packet-bricks.git patch-1


From jira at bro-tracker.atlassian.net  Tue Sep 16 08:18:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Tue, 16 Sep 2014 10:18:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1252) topic/jsiwek/missing-plugin
In-Reply-To: <JIRA.16000.1410880686000@Atlassian.JIRA>
References: <JIRA.16000.1410880686000@Atlassian.JIRA>
	<JIRA.16000.1410880686994@bro-tracker.atlassian.net>
Message-ID: <JIRA.16000.1410880686000.13.1410880687645@Atlassian.JIRA>

Jon Siwek created BIT-1252:
------------------------------

             Summary: topic/jsiwek/missing-plugin
                 Key: BIT-1252
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1252
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: git/master
            Reporter: Jon Siwek
             Fix For: 2.4


Trying to activate a non-existing dynamic plugin gave a hard to understand error message and core dumps:

{noformat}
$ bro Bro::NOPE
error in /Users/jsiwek/Projects/bro/bro/scripts/base/init-bare.bro, line 1: plugin Bro::NOPE is not available
internal error in /Users/jsiwek/Projects/bro/bro/build/scripts/base/bif/const.bif.bro, line 33: internal variable state_dir missing
Abort trap: 6
{noformat}

It also seems that {{bro -N Bro::NOPE}}, which is used by the {{testing/scripts/has-writer}} helper script, is interpreted as a request to activate a dynamic plugin instead of just query if a plugin exists w/ that name.  Not sure the intention: I just changed the helper script to grep the output of {{bro -N}} instead.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Tue Sep 16 08:19:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Tue, 16 Sep 2014 10:19:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1252) topic/jsiwek/missing-plugin
In-Reply-To: <JIRA.16000.1410880686000@Atlassian.JIRA>
References: <JIRA.16000.1410880686000@Atlassian.JIRA>
	<JIRA.16000.1410880686994@bro-tracker.atlassian.net>
Message-ID: <JIRA.16000.1410880686000.14.1410880747632@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1252:
---------------------------
    Status: Merge Request  (was: Open)

> topic/jsiwek/missing-plugin
> ---------------------------
>
>                 Key: BIT-1252
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1252
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>            Reporter: Jon Siwek
>             Fix For: 2.4
>
>
> Trying to activate a non-existing dynamic plugin gave a hard to understand error message and core dumps:
> {noformat}
> $ bro Bro::NOPE
> error in /Users/jsiwek/Projects/bro/bro/scripts/base/init-bare.bro, line 1: plugin Bro::NOPE is not available
> internal error in /Users/jsiwek/Projects/bro/bro/build/scripts/base/bif/const.bif.bro, line 33: internal variable state_dir missing
> Abort trap: 6
> {noformat}
> It also seems that {{bro -N Bro::NOPE}}, which is used by the {{testing/scripts/has-writer}} helper script, is interpreted as a request to activate a dynamic plugin instead of just query if a plugin exists w/ that name.  Not sure the intention: I just changed the helper script to grep the output of {{bro -N}} instead.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From noreply at bro.org  Wed Sep 17 00:00:17 2014
From: noreply at bro.org (Merge Tracker)
Date: Wed, 17 Sep 2014 00:00:17 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409170700.s8H70Hqk001602@bro-ids.icir.org>


Open Merge Requests
===================

ID            Component    Reporter    Assignee    Updated       For Version  Priority    Summary
------------  -----------  ----------  ----------  ----------  -------------  ----------  -------------------------------
BIT-1252 [1]  Bro          Jon Siwek   -           2014-09-16            2.4  Normal      topic/jsiwek/missing-plugin [2]


Open Fastpath Commits
======================

Commit       Component    Author         Date        Summary
-----------  -----------  -------------  ----------  --------------------------------------------------------
d226fef [3]  bro          Daniel Thayer  2014-09-16  Fixed some "make doc" warnings caused by reST formatting


Open GitHub Pull Requests
=========================

Issue    Component      User              Updated     Title
-------  -------------  ----------------  ----------  ---------------------------------------------
#14 [4]  bro            cmavr8 [5]        2014-09-16  Minor corrections in doc/scripting readme [6]
#1 [7]   cmake          dcode [8]         2014-09-17  Added RPM metadata [9]
#2 [10]  packet-bricks  asifjamshed [11]  2014-09-13  typo [12]
#1 [13]  packet-bricks  asifjamshed [14]  2014-09-13  Fixing minor typos [15]


[1]   BIT-1252                     https://bro-tracker.atlassian.net/browse/BIT-1252
[2]   missing-plugin               https://github.com/bro/bro/tree/topic/jsiwek/missing-plugin
[3]   d226fef                      https://github.com/bro/bro/commit/d226fef723325540340c4169884ad28ff8a33290
[4]   Pull Request #14             https://github.com/bro/bro/pull/14
[5]   cmavr8                       https://github.com/cmavr8
[6]   Merge Pull Request #14 with  git pull --no-ff --no-commit https://github.com/cmavr8/bro.git master
[7]   Pull Request #1              https://github.com/bro/cmake/pull/1
[8]   dcode                        https://github.com/dcode
[9]   Merge Pull Request #1 with   git pull --no-ff --no-commit https://github.com/dcode/cmake.git patch-1
[10]  Pull Request #2              https://github.com/bro/packet-bricks/pull/2
[11]  asifjamshed                  https://github.com/asifjamshed
[12]  Merge Pull Request #2 with   git pull --no-ff --no-commit https://github.com/asifjamshed/packet-bricks.git master
[13]  Pull Request #1              https://github.com/bro/packet-bricks/pull/1
[14]  asifjamshed                  https://github.com/asifjamshed
[15]  Merge Pull Request #1 with   git pull --no-ff --no-commit https://github.com/asifjamshed/packet-bricks.git patch-1


From jira at bro-tracker.atlassian.net  Wed Sep 17 05:25:07 2014
From: jira at bro-tracker.atlassian.net (Larry Leviton (JIRA))
Date: Wed, 17 Sep 2014 07:25:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1253) Bro 2.3 - 2.3.1 manager dieing on Bivio
	hardware
In-Reply-To: <JIRA.16001.1410956695000@Atlassian.JIRA>
References: <JIRA.16001.1410956695000@Atlassian.JIRA>
	<JIRA.16001.1410956695631@bro-tracker.atlassian.net>
Message-ID: <JIRA.16001.1410956695000.15.1410956707659@Atlassian.JIRA>

Larry Leviton created BIT-1253:
----------------------------------

             Summary: Bro 2.3 - 2.3.1 manager dieing on Bivio hardware
                 Key: BIT-1253
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1253
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: 2.3
         Environment: Bro 2.3 and Bro 2.3.1 
bivio hardwareLinux CPU.2.6.31-45 has curl 7.36 gperftools 2.2 flex 2.5.39 bison 3.0.2 libpcap 1.1 swig 2.0.8
            Reporter: Larry Leviton


After starting bro up, the bro manager crashes in less than 60 seconds.
Thanks for any help you can give.

Sent stack trace to vendor (at bottom), and here was their response:

Comment(s):	Hello Larry,

We have duplicated a crash in our lab setup that seems to be identical to that experienced by you. The code has changed quite a bit from 2.1 to 2.3.1, and we suspect a bug was introduced.

What is going on, seems to be that a writer thread is being terminated, and the destructor for the Ascii writer is called eventually. However, the destructor code does some checks and finds out that proper cleanup has not been done, so it aborts. This does not seem to be due to any library incompatibility, and looks more like maybe a race condition was introduced.

Since you knows the Bro developers, can you please ask them to take a look this and get back to us? We think it requires their expertise at this point.

Thank You,

Hassan.

 	
Bivio Case Information:	
Bivio Case #:	4566243	
Date Created:	9/02/2014 08:02 AM PDT	

Stack trace below:
GNU gdb (GDB) Fedora (6.8.50.20090302-40.fc11) Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "ppc-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
backtrace
[New Thread 25501]
[New Thread 25328]
[New Thread 25378]
[New Thread 25379]
[New Thread 25380]
[New Thread 25381]
[New Thread 25382]
[New Thread 25383]
[New Thread 25384]
[New Thread 25385]
[New Thread 25386]
[New Thread 25389]
[New Thread 25442]

warning: Can't read pathname for load map: Input/output error.
Missing separate debuginfo for /usr/local/lib/libz.so.1
Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a2/0a0d1fc0d48c2a303af1417ccc03308b9de04a
Missing separate debuginfo for /usr/local/lib/libtcmalloc.so.4
Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/27/eaf56bc64810920d55b9530156c1e8ffbfd43e
Missing separate debuginfo for /usr/local/lib/libcurl.so.4
Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a7/9a2cebb4abc156495ec0806b1c18015c8eba01
Reading symbols from /usr/lib/libpcap.so.1...done.
Loaded symbols for /usr/lib/libpcap.so.1 Reading symbols from /usr/lib/libssl.so.10...done.
Loaded symbols for /usr/lib/libssl.so.10 Reading symbols from /usr/lib/libcrypto.so.10...done.
Loaded symbols for /usr/lib/libcrypto.so.10 Reading symbols from /usr/lib/libbind.so.4...done.
Loaded symbols for /usr/lib/libbind.so.4 Reading symbols from /usr/local/lib/libz.so.1...done.
Loaded symbols for /usr/local/lib/libz.so.1 Reading symbols from /usr/local/lib/libtcmalloc.so.4...done.
Loaded symbols for /usr/local/lib/libtcmalloc.so.4 Reading symbols from /usr/local/lib/libcurl.so.4...done.
Loaded symbols for /usr/local/lib/libcurl.so.4 Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /usr/lib/libstdc++.so.6...done.
Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /usr/lib/libzcp.so...done.
Loaded symbols for /usr/lib/libzcp.so
Reading symbols from /lib/libgssapi_krb5.so.2...done.
Loaded symbols for /lib/libgssapi_krb5.so.2 Reading symbols from /lib/libkrb5.so.3...done.
Loaded symbols for /lib/libkrb5.so.3
Reading symbols from /lib/libcom_err.so.2...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /lib/libk5crypto.so.3...done.
Loaded symbols for /lib/libk5crypto.so.3 Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/librt.so.1...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/ld.so.1...done.
Loaded symbols for /lib/ld.so.1
Reading symbols from /lib/libbvsp.so...done.
Loaded symbols for /lib/libbvsp.so
Reading symbols from /lib/libbcon.so...done.
Loaded symbols for /lib/libbcon.so
Reading symbols from /lib/libkrb5support.so.0...done.
Loaded symbols for /lib/libkrb5support.so.0 Reading symbols from /lib/libkeyutils.so.1...done.
Loaded symbols for /lib/libkeyutils.so.1 Reading symbols from /usr/lib/libxml2.so.2...done.
Loaded symbols for /usr/lib/libxml2.so.2 Reading symbols from /lib/libhmlibs.so...done.
Loaded symbols for /lib/libhmlibs.so
Reading symbols from /lib/libhmolddb.so...done.
Loaded symbols for /lib/libhmolddb.so
Reading symbols from /lib/libcf.so...done.
Loaded symbols for /lib/libcf.so
Reading symbols from /lib/libbvsep.so...done.
Loaded symbols for /lib/libbvsep.so
Reading symbols from /usr/lib/libnrddi.so...done.
Loaded symbols for /usr/lib/libnrddi.so
Reading symbols from /lib/libselinux.so.1...done.
Loaded symbols for /lib/libselinux.so.1
Core was generated by `/var/tmp/bro/spool/tmp/bro -U .status -p broctl -p broctl-live -p local -p mana'.
Program terminated with signal 6, Aborted.
#0  0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56	../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
	in ../nptl/sysdeps/unix/sysv/linux/raise.c
Missing separate debuginfos, use: debuginfo-install e2fsprogs-libs-1.41.9-2.fc11.ppc glibc-2.17-4.fc11.ppc keyutils-libs-1.2-5.fc11.ppc krb5-libs-1.9.3-1.fc11.ppc libbind-6.0-1.fc11.ppc libgcc-4.4.1-2.fc11.ppc libselinux-2.0.80-1.fc11.ppc libstdc++-4.4.1-2.fc11.ppc libxml2-2.7.6-1.fc11.ppc openssl-libs-1.0.1e-37.fc11.1.ppc
(gdb) backtrace
#0  0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x0f6d0de0 in *__GI_abort () at abort.c:90
#2  0x1024be70 in logging::writer::Ascii::~Ascii (this=0x11a87200, __in_chrg=<value optimized out>)
    at /bivio/scsi/b/levitonl/bro-2.3.1/src/logging/writers/Ascii.cc:186
#3  0x10236b70 in threading::Manager::Process (this=0x10dae180)
    at /bivio/scsi/b/levitonl/bro-2.3.1/src/threading/Manager.cc:171
#4  0x101a5400 in net_run () at /bivio/scsi/b/levitonl/bro-2.3.1/src/Net.cc:389
#5  0x100f7554 in main (argc=<value optimized out>, argv=<value optimized out>)
    at /bivio/scsi/b/levitonl/bro-2.3.1/src/main.cc:1165
Current language:  auto; currently minimal
(gdb)
#0  0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x0f6d0de0 in *__GI_abort () at abort.c:90
#2  0x1024be70 in logging::writer::Ascii::~Ascii (this=0x11a87200, __in_chrg=<value optimized out>)
    at /bivio/scsi/b/levitonl/bro-2.3.1/src/logging/writers/Ascii.cc:186
#3  0x10236b70 in threading::Manager::Process (this=0x10dae180)
    at /bivio/scsi/b/levitonl/bro-2.3.1/src/threading/Manager.cc:171
#4  0x101a5400 in net_run () at /bivio/scsi/b/levitonl/bro-2.3.1/src/Net.cc:389
#5  0x100f7554 in main (argc=<value optimized out>, argv=<value optimized out>)
    at /bivio/scsi/b/levitonl/bro-2.3.1/src/main.cc:1165
(gdb) quit




--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Wed Sep 17 06:01:08 2014
From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA))
Date: Wed, 17 Sep 2014 08:01:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1253) Bro 2.3 - 2.3.1 manager dieing on Bivio
	hardware
In-Reply-To: <JIRA.16001.1410956695000@Atlassian.JIRA>
References: <JIRA.16001.1410956695000@Atlassian.JIRA>
	<JIRA.16001.1410956695631@bro-tracker.atlassian.net>
Message-ID: <JIRA.16001.1410956695000.17.1410958868203@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18102#comment-18102 ] 

Johanna Amann commented on BIT-1253:
------------------------------------

If I interpret the backtrace correctly, the thread manager tries to delete the instance before the DoFinish()-function has been called (which triggers an internal error since it should be impossible).

Is there by any chance any output in reporter.log just before the crash?

If no, would it be possible to recompile Bro with --enable-debug and run bro with the option "-B threading" and attach (at least the end) of the resulting debug.log here?

> Bro 2.3 - 2.3.1 manager dieing on Bivio hardware
> ------------------------------------------------
>
>                 Key: BIT-1253
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1253
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: 2.3
>         Environment: Bro 2.3 and Bro 2.3.1 
> bivio hardwareLinux CPU.2.6.31-45 has curl 7.36 gperftools 2.2 flex 2.5.39 bison 3.0.2 libpcap 1.1 swig 2.0.8
>            Reporter: Larry Leviton
>
> After starting bro up, the bro manager crashes in less than 60 seconds.
> Thanks for any help you can give.
> Sent stack trace to vendor (at bottom), and here was their response:
> Comment(s):	Hello Larry,
> We have duplicated a crash in our lab setup that seems to be identical to that experienced by you. The code has changed quite a bit from 2.1 to 2.3.1, and we suspect a bug was introduced.
> What is going on, seems to be that a writer thread is being terminated, and the destructor for the Ascii writer is called eventually. However, the destructor code does some checks and finds out that proper cleanup has not been done, so it aborts. This does not seem to be due to any library incompatibility, and looks more like maybe a race condition was introduced.
> Since you knows the Bro developers, can you please ask them to take a look this and get back to us? We think it requires their expertise at this point.
> Thank You,
> Hassan.
>  	
> Bivio Case Information:	
> Bivio Case #:	4566243	
> Date Created:	9/02/2014 08:02 AM PDT	
> Stack trace below:
> GNU gdb (GDB) Fedora (6.8.50.20090302-40.fc11) Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "ppc-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> backtrace
> [New Thread 25501]
> [New Thread 25328]
> [New Thread 25378]
> [New Thread 25379]
> [New Thread 25380]
> [New Thread 25381]
> [New Thread 25382]
> [New Thread 25383]
> [New Thread 25384]
> [New Thread 25385]
> [New Thread 25386]
> [New Thread 25389]
> [New Thread 25442]
> warning: Can't read pathname for load map: Input/output error.
> Missing separate debuginfo for /usr/local/lib/libz.so.1
> Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a2/0a0d1fc0d48c2a303af1417ccc03308b9de04a
> Missing separate debuginfo for /usr/local/lib/libtcmalloc.so.4
> Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/27/eaf56bc64810920d55b9530156c1e8ffbfd43e
> Missing separate debuginfo for /usr/local/lib/libcurl.so.4
> Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a7/9a2cebb4abc156495ec0806b1c18015c8eba01
> Reading symbols from /usr/lib/libpcap.so.1...done.
> Loaded symbols for /usr/lib/libpcap.so.1 Reading symbols from /usr/lib/libssl.so.10...done.
> Loaded symbols for /usr/lib/libssl.so.10 Reading symbols from /usr/lib/libcrypto.so.10...done.
> Loaded symbols for /usr/lib/libcrypto.so.10 Reading symbols from /usr/lib/libbind.so.4...done.
> Loaded symbols for /usr/lib/libbind.so.4 Reading symbols from /usr/local/lib/libz.so.1...done.
> Loaded symbols for /usr/local/lib/libz.so.1 Reading symbols from /usr/local/lib/libtcmalloc.so.4...done.
> Loaded symbols for /usr/local/lib/libtcmalloc.so.4 Reading symbols from /usr/local/lib/libcurl.so.4...done.
> Loaded symbols for /usr/local/lib/libcurl.so.4 Reading symbols from /lib/libpthread.so.0...done.
> Loaded symbols for /lib/libpthread.so.0
> Reading symbols from /lib/libdl.so.2...done.
> Loaded symbols for /lib/libdl.so.2
> Reading symbols from /usr/lib/libstdc++.so.6...done.
> Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from /lib/libm.so.6...done.
> Loaded symbols for /lib/libm.so.6
> Reading symbols from /lib/libgcc_s.so.1...done.
> Loaded symbols for /lib/libgcc_s.so.1
> Reading symbols from /lib/libc.so.6...done.
> Loaded symbols for /lib/libc.so.6
> Reading symbols from /usr/lib/libzcp.so...done.
> Loaded symbols for /usr/lib/libzcp.so
> Reading symbols from /lib/libgssapi_krb5.so.2...done.
> Loaded symbols for /lib/libgssapi_krb5.so.2 Reading symbols from /lib/libkrb5.so.3...done.
> Loaded symbols for /lib/libkrb5.so.3
> Reading symbols from /lib/libcom_err.so.2...done.
> Loaded symbols for /lib/libcom_err.so.2
> Reading symbols from /lib/libk5crypto.so.3...done.
> Loaded symbols for /lib/libk5crypto.so.3 Reading symbols from /lib/libresolv.so.2...done.
> Loaded symbols for /lib/libresolv.so.2
> Reading symbols from /lib/librt.so.1...done.
> Loaded symbols for /lib/librt.so.1
> Reading symbols from /lib/ld.so.1...done.
> Loaded symbols for /lib/ld.so.1
> Reading symbols from /lib/libbvsp.so...done.
> Loaded symbols for /lib/libbvsp.so
> Reading symbols from /lib/libbcon.so...done.
> Loaded symbols for /lib/libbcon.so
> Reading symbols from /lib/libkrb5support.so.0...done.
> Loaded symbols for /lib/libkrb5support.so.0 Reading symbols from /lib/libkeyutils.so.1...done.
> Loaded symbols for /lib/libkeyutils.so.1 Reading symbols from /usr/lib/libxml2.so.2...done.
> Loaded symbols for /usr/lib/libxml2.so.2 Reading symbols from /lib/libhmlibs.so...done.
> Loaded symbols for /lib/libhmlibs.so
> Reading symbols from /lib/libhmolddb.so...done.
> Loaded symbols for /lib/libhmolddb.so
> Reading symbols from /lib/libcf.so...done.
> Loaded symbols for /lib/libcf.so
> Reading symbols from /lib/libbvsep.so...done.
> Loaded symbols for /lib/libbvsep.so
> Reading symbols from /usr/lib/libnrddi.so...done.
> Loaded symbols for /usr/lib/libnrddi.so
> Reading symbols from /lib/libselinux.so.1...done.
> Loaded symbols for /lib/libselinux.so.1
> Core was generated by `/var/tmp/bro/spool/tmp/bro -U .status -p broctl -p broctl-live -p local -p mana'.
> Program terminated with signal 6, Aborted.
> #0  0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> 56	../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> 	in ../nptl/sysdeps/unix/sysv/linux/raise.c
> Missing separate debuginfos, use: debuginfo-install e2fsprogs-libs-1.41.9-2.fc11.ppc glibc-2.17-4.fc11.ppc keyutils-libs-1.2-5.fc11.ppc krb5-libs-1.9.3-1.fc11.ppc libbind-6.0-1.fc11.ppc libgcc-4.4.1-2.fc11.ppc libselinux-2.0.80-1.fc11.ppc libstdc++-4.4.1-2.fc11.ppc libxml2-2.7.6-1.fc11.ppc openssl-libs-1.0.1e-37.fc11.1.ppc
> (gdb) backtrace
> #0  0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1  0x0f6d0de0 in *__GI_abort () at abort.c:90
> #2  0x1024be70 in logging::writer::Ascii::~Ascii (this=0x11a87200, __in_chrg=<value optimized out>)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/logging/writers/Ascii.cc:186
> #3  0x10236b70 in threading::Manager::Process (this=0x10dae180)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/threading/Manager.cc:171
> #4  0x101a5400 in net_run () at /bivio/scsi/b/levitonl/bro-2.3.1/src/Net.cc:389
> #5  0x100f7554 in main (argc=<value optimized out>, argv=<value optimized out>)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/main.cc:1165
> Current language:  auto; currently minimal
> (gdb)
> #0  0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1  0x0f6d0de0 in *__GI_abort () at abort.c:90
> #2  0x1024be70 in logging::writer::Ascii::~Ascii (this=0x11a87200, __in_chrg=<value optimized out>)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/logging/writers/Ascii.cc:186
> #3  0x10236b70 in threading::Manager::Process (this=0x10dae180)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/threading/Manager.cc:171
> #4  0x101a5400 in net_run () at /bivio/scsi/b/levitonl/bro-2.3.1/src/Net.cc:389
> #5  0x100f7554 in main (argc=<value optimized out>, argv=<value optimized out>)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/main.cc:1165
> (gdb) quit



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Wed Sep 17 06:18:08 2014
From: jira at bro-tracker.atlassian.net (Larry Leviton (JIRA))
Date: Wed, 17 Sep 2014 08:18:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1253) Bro 2.3 - 2.3.1 manager dieing on Bivio
	hardware
In-Reply-To: <JIRA.16001.1410956695000@Atlassian.JIRA>
References: <JIRA.16001.1410956695000@Atlassian.JIRA>
	<JIRA.16001.1410956695631@bro-tracker.atlassian.net>
Message-ID: <JIRA.16001.1410956695000.19.1410959888443@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18103#comment-18103 ] 

Larry Leviton commented on BIT-1253:
------------------------------------

Ok, I'll try and get it done today, but am leaving for APPSEC USA, so good
chance I won't get to it till Monday or Tuesday.

Thanks for the prompt response.

BTW: I also have another unrelated stack trace, but will wait to try and
run it with debug on and run it again, and then I will submit it next week
(Also didn't try it in 2.3.1).

#0  nb_dns_fd (nd=0x0) at /bivio/scsi/b/levitonl/bro-2.3/src/nb_dns.c:177

#1  0x1012546c in DNS_Mgr::GetFds (this=<value optimized out>, read=<value
optim ized out>, write=<value optimized out>,

    except=<value optimized out>) at
/bivio/scsi/b/levitonl/bro-2.3/src/DNS_Mgr.

cc:1222

#2  0x10198f28 in IOSourceRegistry::FindSoonest (this=0x1058596c,
ts=0xbfce0fd8)

    at /bivio/scsi/b/levitonl/bro-2.3/src/IOSource.cc:98

#3  0x101a2d8c in net_run () at
/bivio/scsi/b/levitonl/bro-2.3/src/Net.cc:370

#4  0x100f7228 in main (argc=<value optimized out>, argv=<value optimized
out>)

    at /bivio/scsi/b/levitonl/bro-2.3/src/main.cc:1165




On Wed, Sep 17, 2014 at 9:01 AM, Johanna Amann (JIRA) <



> Bro 2.3 - 2.3.1 manager dieing on Bivio hardware
> ------------------------------------------------
>
>                 Key: BIT-1253
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1253
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: 2.3
>         Environment: Bro 2.3 and Bro 2.3.1 
> bivio hardwareLinux CPU.2.6.31-45 has curl 7.36 gperftools 2.2 flex 2.5.39 bison 3.0.2 libpcap 1.1 swig 2.0.8
>            Reporter: Larry Leviton
>
> After starting bro up, the bro manager crashes in less than 60 seconds.
> Thanks for any help you can give.
> Sent stack trace to vendor (at bottom), and here was their response:
> Comment(s):	Hello Larry,
> We have duplicated a crash in our lab setup that seems to be identical to that experienced by you. The code has changed quite a bit from 2.1 to 2.3.1, and we suspect a bug was introduced.
> What is going on, seems to be that a writer thread is being terminated, and the destructor for the Ascii writer is called eventually. However, the destructor code does some checks and finds out that proper cleanup has not been done, so it aborts. This does not seem to be due to any library incompatibility, and looks more like maybe a race condition was introduced.
> Since you knows the Bro developers, can you please ask them to take a look this and get back to us? We think it requires their expertise at this point.
> Thank You,
> Hassan.
>  	
> Bivio Case Information:	
> Bivio Case #:	4566243	
> Date Created:	9/02/2014 08:02 AM PDT	
> Stack trace below:
> GNU gdb (GDB) Fedora (6.8.50.20090302-40.fc11) Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "ppc-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> backtrace
> [New Thread 25501]
> [New Thread 25328]
> [New Thread 25378]
> [New Thread 25379]
> [New Thread 25380]
> [New Thread 25381]
> [New Thread 25382]
> [New Thread 25383]
> [New Thread 25384]
> [New Thread 25385]
> [New Thread 25386]
> [New Thread 25389]
> [New Thread 25442]
> warning: Can't read pathname for load map: Input/output error.
> Missing separate debuginfo for /usr/local/lib/libz.so.1
> Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a2/0a0d1fc0d48c2a303af1417ccc03308b9de04a
> Missing separate debuginfo for /usr/local/lib/libtcmalloc.so.4
> Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/27/eaf56bc64810920d55b9530156c1e8ffbfd43e
> Missing separate debuginfo for /usr/local/lib/libcurl.so.4
> Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a7/9a2cebb4abc156495ec0806b1c18015c8eba01
> Reading symbols from /usr/lib/libpcap.so.1...done.
> Loaded symbols for /usr/lib/libpcap.so.1 Reading symbols from /usr/lib/libssl.so.10...done.
> Loaded symbols for /usr/lib/libssl.so.10 Reading symbols from /usr/lib/libcrypto.so.10...done.
> Loaded symbols for /usr/lib/libcrypto.so.10 Reading symbols from /usr/lib/libbind.so.4...done.
> Loaded symbols for /usr/lib/libbind.so.4 Reading symbols from /usr/local/lib/libz.so.1...done.
> Loaded symbols for /usr/local/lib/libz.so.1 Reading symbols from /usr/local/lib/libtcmalloc.so.4...done.
> Loaded symbols for /usr/local/lib/libtcmalloc.so.4 Reading symbols from /usr/local/lib/libcurl.so.4...done.
> Loaded symbols for /usr/local/lib/libcurl.so.4 Reading symbols from /lib/libpthread.so.0...done.
> Loaded symbols for /lib/libpthread.so.0
> Reading symbols from /lib/libdl.so.2...done.
> Loaded symbols for /lib/libdl.so.2
> Reading symbols from /usr/lib/libstdc++.so.6...done.
> Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from /lib/libm.so.6...done.
> Loaded symbols for /lib/libm.so.6
> Reading symbols from /lib/libgcc_s.so.1...done.
> Loaded symbols for /lib/libgcc_s.so.1
> Reading symbols from /lib/libc.so.6...done.
> Loaded symbols for /lib/libc.so.6
> Reading symbols from /usr/lib/libzcp.so...done.
> Loaded symbols for /usr/lib/libzcp.so
> Reading symbols from /lib/libgssapi_krb5.so.2...done.
> Loaded symbols for /lib/libgssapi_krb5.so.2 Reading symbols from /lib/libkrb5.so.3...done.
> Loaded symbols for /lib/libkrb5.so.3
> Reading symbols from /lib/libcom_err.so.2...done.
> Loaded symbols for /lib/libcom_err.so.2
> Reading symbols from /lib/libk5crypto.so.3...done.
> Loaded symbols for /lib/libk5crypto.so.3 Reading symbols from /lib/libresolv.so.2...done.
> Loaded symbols for /lib/libresolv.so.2
> Reading symbols from /lib/librt.so.1...done.
> Loaded symbols for /lib/librt.so.1
> Reading symbols from /lib/ld.so.1...done.
> Loaded symbols for /lib/ld.so.1
> Reading symbols from /lib/libbvsp.so...done.
> Loaded symbols for /lib/libbvsp.so
> Reading symbols from /lib/libbcon.so...done.
> Loaded symbols for /lib/libbcon.so
> Reading symbols from /lib/libkrb5support.so.0...done.
> Loaded symbols for /lib/libkrb5support.so.0 Reading symbols from /lib/libkeyutils.so.1...done.
> Loaded symbols for /lib/libkeyutils.so.1 Reading symbols from /usr/lib/libxml2.so.2...done.
> Loaded symbols for /usr/lib/libxml2.so.2 Reading symbols from /lib/libhmlibs.so...done.
> Loaded symbols for /lib/libhmlibs.so
> Reading symbols from /lib/libhmolddb.so...done.
> Loaded symbols for /lib/libhmolddb.so
> Reading symbols from /lib/libcf.so...done.
> Loaded symbols for /lib/libcf.so
> Reading symbols from /lib/libbvsep.so...done.
> Loaded symbols for /lib/libbvsep.so
> Reading symbols from /usr/lib/libnrddi.so...done.
> Loaded symbols for /usr/lib/libnrddi.so
> Reading symbols from /lib/libselinux.so.1...done.
> Loaded symbols for /lib/libselinux.so.1
> Core was generated by `/var/tmp/bro/spool/tmp/bro -U .status -p broctl -p broctl-live -p local -p mana'.
> Program terminated with signal 6, Aborted.
> #0  0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> 56	../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> 	in ../nptl/sysdeps/unix/sysv/linux/raise.c
> Missing separate debuginfos, use: debuginfo-install e2fsprogs-libs-1.41.9-2.fc11.ppc glibc-2.17-4.fc11.ppc keyutils-libs-1.2-5.fc11.ppc krb5-libs-1.9.3-1.fc11.ppc libbind-6.0-1.fc11.ppc libgcc-4.4.1-2.fc11.ppc libselinux-2.0.80-1.fc11.ppc libstdc++-4.4.1-2.fc11.ppc libxml2-2.7.6-1.fc11.ppc openssl-libs-1.0.1e-37.fc11.1.ppc
> (gdb) backtrace
> #0  0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1  0x0f6d0de0 in *__GI_abort () at abort.c:90
> #2  0x1024be70 in logging::writer::Ascii::~Ascii (this=0x11a87200, __in_chrg=<value optimized out>)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/logging/writers/Ascii.cc:186
> #3  0x10236b70 in threading::Manager::Process (this=0x10dae180)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/threading/Manager.cc:171
> #4  0x101a5400 in net_run () at /bivio/scsi/b/levitonl/bro-2.3.1/src/Net.cc:389
> #5  0x100f7554 in main (argc=<value optimized out>, argv=<value optimized out>)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/main.cc:1165
> Current language:  auto; currently minimal
> (gdb)
> #0  0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1  0x0f6d0de0 in *__GI_abort () at abort.c:90
> #2  0x1024be70 in logging::writer::Ascii::~Ascii (this=0x11a87200, __in_chrg=<value optimized out>)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/logging/writers/Ascii.cc:186
> #3  0x10236b70 in threading::Manager::Process (this=0x10dae180)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/threading/Manager.cc:171
> #4  0x101a5400 in net_run () at /bivio/scsi/b/levitonl/bro-2.3.1/src/Net.cc:389
> #5  0x100f7554 in main (argc=<value optimized out>, argv=<value optimized out>)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/main.cc:1165
> (gdb) quit



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Wed Sep 17 08:46:07 2014
From: jira at bro-tracker.atlassian.net (Larry Leviton (JIRA))
Date: Wed, 17 Sep 2014 10:46:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1253) Bro 2.3 - 2.3.1 manager dieing on Bivio
	hardware
In-Reply-To: <JIRA.16001.1410956695000@Atlassian.JIRA>
References: <JIRA.16001.1410956695000@Atlassian.JIRA>
	<JIRA.16001.1410956695631@bro-tracker.atlassian.net>
Message-ID: <JIRA.16001.1410956695000.21.1410968767716@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18104#comment-18104 ] 

Larry Leviton commented on BIT-1253:
------------------------------------

Seems to run okay with debug mode on.  No crashes.

The machine I put it on originally had 2.3, and I had DNS logging turned
off (due to stack trace I had sent in first e-mail).    After that, I got
the manager crashes on 2.3 without debug, but the box itself was running
argus at the same time, so I put it on a second standalone machine only
running bro figuring that running both could cause some sort of unique
problem.  So today, I put on 2.3.1 on the machine with argus and recompiled
with debug, and everything is running without crashes.

I am going to compile it with debug on the second machine which is only
running bro.  Hopefully I will get back to you before I leave today.






> Bro 2.3 - 2.3.1 manager dieing on Bivio hardware
> ------------------------------------------------
>
>                 Key: BIT-1253
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1253
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: 2.3
>         Environment: Bro 2.3 and Bro 2.3.1 
> bivio hardwareLinux CPU.2.6.31-45 has curl 7.36 gperftools 2.2 flex 2.5.39 bison 3.0.2 libpcap 1.1 swig 2.0.8
>            Reporter: Larry Leviton
>
> After starting bro up, the bro manager crashes in less than 60 seconds.
> Thanks for any help you can give.
> Sent stack trace to vendor (at bottom), and here was their response:
> Comment(s):	Hello Larry,
> We have duplicated a crash in our lab setup that seems to be identical to that experienced by you. The code has changed quite a bit from 2.1 to 2.3.1, and we suspect a bug was introduced.
> What is going on, seems to be that a writer thread is being terminated, and the destructor for the Ascii writer is called eventually. However, the destructor code does some checks and finds out that proper cleanup has not been done, so it aborts. This does not seem to be due to any library incompatibility, and looks more like maybe a race condition was introduced.
> Since you knows the Bro developers, can you please ask them to take a look this and get back to us? We think it requires their expertise at this point.
> Thank You,
> Hassan.
>  	
> Bivio Case Information:	
> Bivio Case #:	4566243	
> Date Created:	9/02/2014 08:02 AM PDT	
> Stack trace below:
> GNU gdb (GDB) Fedora (6.8.50.20090302-40.fc11) Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "ppc-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> backtrace
> [New Thread 25501]
> [New Thread 25328]
> [New Thread 25378]
> [New Thread 25379]
> [New Thread 25380]
> [New Thread 25381]
> [New Thread 25382]
> [New Thread 25383]
> [New Thread 25384]
> [New Thread 25385]
> [New Thread 25386]
> [New Thread 25389]
> [New Thread 25442]
> warning: Can't read pathname for load map: Input/output error.
> Missing separate debuginfo for /usr/local/lib/libz.so.1
> Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a2/0a0d1fc0d48c2a303af1417ccc03308b9de04a
> Missing separate debuginfo for /usr/local/lib/libtcmalloc.so.4
> Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/27/eaf56bc64810920d55b9530156c1e8ffbfd43e
> Missing separate debuginfo for /usr/local/lib/libcurl.so.4
> Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a7/9a2cebb4abc156495ec0806b1c18015c8eba01
> Reading symbols from /usr/lib/libpcap.so.1...done.
> Loaded symbols for /usr/lib/libpcap.so.1 Reading symbols from /usr/lib/libssl.so.10...done.
> Loaded symbols for /usr/lib/libssl.so.10 Reading symbols from /usr/lib/libcrypto.so.10...done.
> Loaded symbols for /usr/lib/libcrypto.so.10 Reading symbols from /usr/lib/libbind.so.4...done.
> Loaded symbols for /usr/lib/libbind.so.4 Reading symbols from /usr/local/lib/libz.so.1...done.
> Loaded symbols for /usr/local/lib/libz.so.1 Reading symbols from /usr/local/lib/libtcmalloc.so.4...done.
> Loaded symbols for /usr/local/lib/libtcmalloc.so.4 Reading symbols from /usr/local/lib/libcurl.so.4...done.
> Loaded symbols for /usr/local/lib/libcurl.so.4 Reading symbols from /lib/libpthread.so.0...done.
> Loaded symbols for /lib/libpthread.so.0
> Reading symbols from /lib/libdl.so.2...done.
> Loaded symbols for /lib/libdl.so.2
> Reading symbols from /usr/lib/libstdc++.so.6...done.
> Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from /lib/libm.so.6...done.
> Loaded symbols for /lib/libm.so.6
> Reading symbols from /lib/libgcc_s.so.1...done.
> Loaded symbols for /lib/libgcc_s.so.1
> Reading symbols from /lib/libc.so.6...done.
> Loaded symbols for /lib/libc.so.6
> Reading symbols from /usr/lib/libzcp.so...done.
> Loaded symbols for /usr/lib/libzcp.so
> Reading symbols from /lib/libgssapi_krb5.so.2...done.
> Loaded symbols for /lib/libgssapi_krb5.so.2 Reading symbols from /lib/libkrb5.so.3...done.
> Loaded symbols for /lib/libkrb5.so.3
> Reading symbols from /lib/libcom_err.so.2...done.
> Loaded symbols for /lib/libcom_err.so.2
> Reading symbols from /lib/libk5crypto.so.3...done.
> Loaded symbols for /lib/libk5crypto.so.3 Reading symbols from /lib/libresolv.so.2...done.
> Loaded symbols for /lib/libresolv.so.2
> Reading symbols from /lib/librt.so.1...done.
> Loaded symbols for /lib/librt.so.1
> Reading symbols from /lib/ld.so.1...done.
> Loaded symbols for /lib/ld.so.1
> Reading symbols from /lib/libbvsp.so...done.
> Loaded symbols for /lib/libbvsp.so
> Reading symbols from /lib/libbcon.so...done.
> Loaded symbols for /lib/libbcon.so
> Reading symbols from /lib/libkrb5support.so.0...done.
> Loaded symbols for /lib/libkrb5support.so.0 Reading symbols from /lib/libkeyutils.so.1...done.
> Loaded symbols for /lib/libkeyutils.so.1 Reading symbols from /usr/lib/libxml2.so.2...done.
> Loaded symbols for /usr/lib/libxml2.so.2 Reading symbols from /lib/libhmlibs.so...done.
> Loaded symbols for /lib/libhmlibs.so
> Reading symbols from /lib/libhmolddb.so...done.
> Loaded symbols for /lib/libhmolddb.so
> Reading symbols from /lib/libcf.so...done.
> Loaded symbols for /lib/libcf.so
> Reading symbols from /lib/libbvsep.so...done.
> Loaded symbols for /lib/libbvsep.so
> Reading symbols from /usr/lib/libnrddi.so...done.
> Loaded symbols for /usr/lib/libnrddi.so
> Reading symbols from /lib/libselinux.so.1...done.
> Loaded symbols for /lib/libselinux.so.1
> Core was generated by `/var/tmp/bro/spool/tmp/bro -U .status -p broctl -p broctl-live -p local -p mana'.
> Program terminated with signal 6, Aborted.
> #0  0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> 56	../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> 	in ../nptl/sysdeps/unix/sysv/linux/raise.c
> Missing separate debuginfos, use: debuginfo-install e2fsprogs-libs-1.41.9-2.fc11.ppc glibc-2.17-4.fc11.ppc keyutils-libs-1.2-5.fc11.ppc krb5-libs-1.9.3-1.fc11.ppc libbind-6.0-1.fc11.ppc libgcc-4.4.1-2.fc11.ppc libselinux-2.0.80-1.fc11.ppc libstdc++-4.4.1-2.fc11.ppc libxml2-2.7.6-1.fc11.ppc openssl-libs-1.0.1e-37.fc11.1.ppc
> (gdb) backtrace
> #0  0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1  0x0f6d0de0 in *__GI_abort () at abort.c:90
> #2  0x1024be70 in logging::writer::Ascii::~Ascii (this=0x11a87200, __in_chrg=<value optimized out>)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/logging/writers/Ascii.cc:186
> #3  0x10236b70 in threading::Manager::Process (this=0x10dae180)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/threading/Manager.cc:171
> #4  0x101a5400 in net_run () at /bivio/scsi/b/levitonl/bro-2.3.1/src/Net.cc:389
> #5  0x100f7554 in main (argc=<value optimized out>, argv=<value optimized out>)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/main.cc:1165
> Current language:  auto; currently minimal
> (gdb)
> #0  0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1  0x0f6d0de0 in *__GI_abort () at abort.c:90
> #2  0x1024be70 in logging::writer::Ascii::~Ascii (this=0x11a87200, __in_chrg=<value optimized out>)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/logging/writers/Ascii.cc:186
> #3  0x10236b70 in threading::Manager::Process (this=0x10dae180)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/threading/Manager.cc:171
> #4  0x101a5400 in net_run () at /bivio/scsi/b/levitonl/bro-2.3.1/src/Net.cc:389
> #5  0x100f7554 in main (argc=<value optimized out>, argv=<value optimized out>)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/main.cc:1165
> (gdb) quit



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From noreply at bro.org  Thu Sep 18 00:00:19 2014
From: noreply at bro.org (Merge Tracker)
Date: Thu, 18 Sep 2014 00:00:19 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409180700.s8I70JGb024721@bro-ids.icir.org>


Open Merge Requests
===================

ID            Component    Reporter    Assignee    Updated       For Version  Priority    Summary
------------  -----------  ----------  ----------  ----------  -------------  ----------  -------------------------------
BIT-1252 [1]  Bro          Jon Siwek   -           2014-09-16            2.4  Normal      topic/jsiwek/missing-plugin [2]


Open GitHub Pull Requests
=========================

Issue    Component      User                               Updated     Title
-------  -------------  ---------------------------------  ----------  ------------------------------
#15 [3]  bro            drsirmrpresidentfathercharles [4]  2014-09-18  Rename README to README.md [5]
#1 [6]   cmake          dcode [7]                          2014-09-17  Added RPM metadata [8]
#2 [9]   packet-bricks  asifjamshed [10]                   2014-09-13  typo [11]
#1 [12]  packet-bricks  asifjamshed [13]                   2014-09-13  Fixing minor typos [14]


[1]   BIT-1252                       https://bro-tracker.atlassian.net/browse/BIT-1252
[2]   missing-plugin                 https://github.com/bro/bro/tree/topic/jsiwek/missing-plugin
[3]   Pull Request #15               https://github.com/bro/bro/pull/15
[4]   drsirmrpresidentfathercharles  https://github.com/drsirmrpresidentfathercharles
[5]   Merge Pull Request #15 with    git pull --no-ff --no-commit https://github.com/drsirmrpresidentfathercharles/bro.git master
[6]   Pull Request #1                https://github.com/bro/cmake/pull/1
[7]   dcode                          https://github.com/dcode
[8]   Merge Pull Request #1 with     git pull --no-ff --no-commit https://github.com/dcode/cmake.git patch-1
[9]   Pull Request #2                https://github.com/bro/packet-bricks/pull/2
[10]  asifjamshed                    https://github.com/asifjamshed
[11]  Merge Pull Request #2 with     git pull --no-ff --no-commit https://github.com/asifjamshed/packet-bricks.git master
[12]  Pull Request #1                https://github.com/bro/packet-bricks/pull/1
[13]  asifjamshed                    https://github.com/asifjamshed
[14]  Merge Pull Request #1 with     git pull --no-ff --no-commit https://github.com/asifjamshed/packet-bricks.git patch-1


From jira at bro-tracker.atlassian.net  Thu Sep 18 04:58:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Thu, 18 Sep 2014 06:58:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1254) file analysis framework sometimes
 returns hashes despite missing packets
In-Reply-To: <JIRA.16002.1411041471000@Atlassian.JIRA>
References: <JIRA.16002.1411041471000@Atlassian.JIRA>
	<JIRA.16002.1411041471395@bro-tracker.atlassian.net>
Message-ID: <JIRA.16002.1411041471000.22.1411041487651@Atlassian.JIRA>

Jimmy Jones created BIT-1254:
--------------------------------

             Summary: file analysis framework sometimes returns hashes despite missing packets
                 Key: BIT-1254
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1254
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: 2.3, git/master
         Environment: CentOS 6
            Reporter: Jimmy Jones
         Attachments: sample-3streams-hole.pcap

Putting the attached sample (3 streams, each with missing packets) though the file analysis framework, in files.log I see hashes for one streams but not the other 2. Should I get any hashes if there are missing packets?

bro -r sample-3streams-hole.pcap frameworks/files/hash-all-files.bro



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 18 07:22:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Thu, 18 Sep 2014 09:22:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1255) TCP reassembly issue
In-Reply-To: <JIRA.16003.1411050122000@Atlassian.JIRA>
References: <JIRA.16003.1411050122000@Atlassian.JIRA>
	<JIRA.16003.1411050122055@bro-tracker.atlassian.net>
Message-ID: <JIRA.16003.1411050122000.23.1411050127655@Atlassian.JIRA>

Jimmy Jones created BIT-1255:
--------------------------------

             Summary: TCP reassembly issue
                 Key: BIT-1255
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1255
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: 2.3, git/master
         Environment: CentOS 6
            Reporter: Jimmy Jones
         Attachments: out.pcap

Been testing bro with some messy (but valid) TCP streams, using docker and netem (happy to upload a gist if people are interested).

The attached file reassembles correctly in wireshark, but bro only gives the first 4069 bytes when extracted with the file analysis framework, and obviously the wrong hash (md5 is the URI).



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 18 08:49:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Thu, 18 Sep 2014 10:49:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1256) file_state_remove event after bro_done
In-Reply-To: <JIRA.16004.1411055301000@Atlassian.JIRA>
References: <JIRA.16004.1411055301000@Atlassian.JIRA>
	<JIRA.16004.1411055301220@bro-tracker.atlassian.net>
Message-ID: <JIRA.16004.1411055301000.24.1411055347649@Atlassian.JIRA>

Jimmy Jones created BIT-1256:
--------------------------------

             Summary: file_state_remove event after bro_done
                 Key: BIT-1256
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1256
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: 2.3, git/master
         Environment: CentOS 6
            Reporter: Jimmy Jones
         Attachments: fa.bro, weird8.pcap

Using the attached bro script (just prints in bro_done and file_state_remove events) and capture, the bro_done event is fired before file_state_remove.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 18 11:22:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Thu, 18 Sep 2014 13:22:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1254) file analysis framework sometimes
 returns hashes despite missing packets
In-Reply-To: <JIRA.16002.1411041471000@Atlassian.JIRA>
References: <JIRA.16002.1411041471000@Atlassian.JIRA>
	<JIRA.16002.1411041471395@bro-tracker.atlassian.net>
Message-ID: <JIRA.16002.1411041471000.26.1411064527747@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18105#comment-18105 ] 

Jon Siwek commented on BIT-1254:
--------------------------------

If you mean missing content, then the idea was to abort the hashing for the file since it would be incorrect.  If you mean that you know some particular packets are missing (maybe because you manually modified the capture), then it depends on if the missing packets actually created gaps -- do you know if that's true?  Looking quickly in wireshark: it also doesn't seem to report missing bytes in that stream, but does in the other two, so maybe the missing packets were duplicates or control packets?

> file analysis framework sometimes returns hashes despite missing packets
> ------------------------------------------------------------------------
>
>                 Key: BIT-1254
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1254
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: sample-3streams-hole.pcap
>
>
> Putting the attached sample (3 streams, each with missing packets) though the file analysis framework, in files.log I see hashes for one streams but not the other 2. Should I get any hashes if there are missing packets?
> bro -r sample-3streams-hole.pcap frameworks/files/hash-all-files.bro



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 18 12:03:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Thu, 18 Sep 2014 14:03:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1255) TCP reassembly issue
In-Reply-To: <JIRA.16003.1411050122000@Atlassian.JIRA>
References: <JIRA.16003.1411050122000@Atlassian.JIRA>
	<JIRA.16003.1411050122055@bro-tracker.atlassian.net>
Message-ID: <JIRA.16003.1411050122000.28.1411066987719@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18106#comment-18106 ] 

Jon Siwek commented on BIT-1255:
--------------------------------

There's a bit of a hint in weird.log: "above_hole_data_without_any_acks".  A description can be found at:

https://www.bro.org/sphinx/scripts/base/init-bare.bro.html?highlight=above_hole#id-tcp_max_above_hole_without_any_acks

When it says Bro will "give up", that's pretty literal -- no extra efforts are taken with the analysis and whatever's been done up to that point is basically what you get.  However, you can redef that option to tune Bro's behavior.  For example, try running that trace w/ {{redef tcp_max_above_hole_without_any_acks=10000;}}.  With git/master, I get the right md5.  Generally, the goal of this is to prevent Bro from buffering unreasonable amounts of data as it waits to fill in a gap in the TCP stream (which isn't guaranteed to happen).

> TCP reassembly issue
> --------------------
>
>                 Key: BIT-1255
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1255
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: out.pcap
>
>
> Been testing bro with some messy (but valid) TCP streams, using docker and netem (happy to upload a gist if people are interested).
> The attached file reassembles correctly in wireshark, but bro only gives the first 4069 bytes when extracted with the file analysis framework, and obviously the wrong hash (md5 is the URI).



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 18 12:06:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Thu, 18 Sep 2014 14:06:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1254) file analysis framework sometimes
 returns hashes despite missing packets
In-Reply-To: <JIRA.16002.1411041471000@Atlassian.JIRA>
References: <JIRA.16002.1411041471000@Atlassian.JIRA>
	<JIRA.16002.1411041471395@bro-tracker.atlassian.net>
Message-ID: <JIRA.16002.1411041471000.29.1411067167624@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18107#comment-18107 ] 

Jimmy Jones commented on BIT-1254:
----------------------------------

Sorry, my bad, there aren't any missing packets in tcp.stream == 1 / tcp.port == 48049. It looks to me as if the stream is cut off (without RST or FIN), but has no missing packets up until that point. However there are not as many bytes as the Content-Length indicates, so it is definitly truncated. Should I get a hash if this happens?

Sadly I didn't remove the packets, my tcpdump wasn't set up with a large enough buffer when I was trying to do something else and noticed this!

> file analysis framework sometimes returns hashes despite missing packets
> ------------------------------------------------------------------------
>
>                 Key: BIT-1254
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1254
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: sample-3streams-hole.pcap
>
>
> Putting the attached sample (3 streams, each with missing packets) though the file analysis framework, in files.log I see hashes for one streams but not the other 2. Should I get any hashes if there are missing packets?
> bro -r sample-3streams-hole.pcap frameworks/files/hash-all-files.bro



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 18 12:23:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Thu, 18 Sep 2014 14:23:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1254) file analysis framework sometimes
 returns hashes despite missing packets
In-Reply-To: <JIRA.16002.1411041471000@Atlassian.JIRA>
References: <JIRA.16002.1411041471000@Atlassian.JIRA>
	<JIRA.16002.1411041471395@bro-tracker.atlassian.net>
Message-ID: <JIRA.16002.1411041471000.31.1411068187683@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18108#comment-18108 ] 

Jon Siwek commented on BIT-1254:
--------------------------------

{quote}
However there are not as many bytes as the Content-Length indicates, so it is definitly truncated. Should I get a hash if this happens?
{quote}

I'm kind of inclined to leave it and let the user decide whether to trust the hash in the case that there's no missing_bytes and the total_bytes field is set, but not equal to seen_bytes.  But open to opinions.

> file analysis framework sometimes returns hashes despite missing packets
> ------------------------------------------------------------------------
>
>                 Key: BIT-1254
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1254
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: sample-3streams-hole.pcap
>
>
> Putting the attached sample (3 streams, each with missing packets) though the file analysis framework, in files.log I see hashes for one streams but not the other 2. Should I get any hashes if there are missing packets?
> bro -r sample-3streams-hole.pcap frameworks/files/hash-all-files.bro



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 18 12:32:08 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Thu, 18 Sep 2014 14:32:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1256) file_state_remove event after bro_done
In-Reply-To: <JIRA.16004.1411055301000@Atlassian.JIRA>
References: <JIRA.16004.1411055301000@Atlassian.JIRA>
	<JIRA.16004.1411055301220@bro-tracker.atlassian.net>
Message-ID: <JIRA.16004.1411055301000.33.1411068728165@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1256:
---------------------------
    Fix Version/s: 2.4

> file_state_remove event after bro_done
> --------------------------------------
>
>                 Key: BIT-1256
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1256
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>             Fix For: 2.4
>
>         Attachments: fa.bro, weird8.pcap
>
>
> Using the attached bro script (just prints in bro_done and file_state_remove events) and capture, the bro_done event is fired before file_state_remove.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 18 12:35:08 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Thu, 18 Sep 2014 14:35:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1256) file_state_remove event after bro_done
In-Reply-To: <JIRA.16004.1411055301000@Atlassian.JIRA>
References: <JIRA.16004.1411055301000@Atlassian.JIRA>
	<JIRA.16004.1411055301220@bro-tracker.atlassian.net>
Message-ID: <JIRA.16004.1411055301000.35.1411068908305@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1256:
---------------------------
    Resolution: Fixed
        Status: Closed  (was: Open)

> file_state_remove event after bro_done
> --------------------------------------
>
>                 Key: BIT-1256
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1256
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>             Fix For: 2.4
>
>         Attachments: fa.bro, weird8.pcap
>
>
> Using the attached bro script (just prints in bro_done and file_state_remove events) and capture, the bro_done event is fired before file_state_remove.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 18 12:35:08 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Thu, 18 Sep 2014 14:35:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1256) file_state_remove event after bro_done
In-Reply-To: <JIRA.16004.1411055301000@Atlassian.JIRA>
References: <JIRA.16004.1411055301000@Atlassian.JIRA>
	<JIRA.16004.1411055301220@bro-tracker.atlassian.net>
Message-ID: <JIRA.16004.1411055301000.37.1411068908404@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18109#comment-18109 ] 

Jon Siwek commented on BIT-1256:
--------------------------------

Thanks.  Should be fixed in git/master now.

> file_state_remove event after bro_done
> --------------------------------------
>
>                 Key: BIT-1256
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1256
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>             Fix For: 2.4
>
>         Attachments: fa.bro, weird8.pcap
>
>
> Using the attached bro script (just prints in bro_done and file_state_remove events) and capture, the bro_done event is fired before file_state_remove.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 18 13:04:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Thu, 18 Sep 2014 15:04:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1254) file analysis framework sometimes
 returns hashes despite missing packets
In-Reply-To: <JIRA.16002.1411041471000@Atlassian.JIRA>
References: <JIRA.16002.1411041471000@Atlassian.JIRA>
	<JIRA.16002.1411041471395@bro-tracker.atlassian.net>
Message-ID: <JIRA.16002.1411041471000.38.1411070647626@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18110#comment-18110 ] 

Jimmy Jones commented on BIT-1254:
----------------------------------

I think that's fair, hadn't noticed total_bytes vs. seen_bytes.

> file analysis framework sometimes returns hashes despite missing packets
> ------------------------------------------------------------------------
>
>                 Key: BIT-1254
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1254
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: sample-3streams-hole.pcap
>
>
> Putting the attached sample (3 streams, each with missing packets) though the file analysis framework, in files.log I see hashes for one streams but not the other 2. Should I get any hashes if there are missing packets?
> bro -r sample-3streams-hole.pcap frameworks/files/hash-all-files.bro



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 18 13:08:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Thu, 18 Sep 2014 15:08:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1254) file analysis framework sometimes
 returns hashes despite missing packets
In-Reply-To: <JIRA.16002.1411041471000@Atlassian.JIRA>
References: <JIRA.16002.1411041471000@Atlassian.JIRA>
	<JIRA.16002.1411041471395@bro-tracker.atlassian.net>
Message-ID: <JIRA.16002.1411041471000.39.1411070887623@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jimmy Jones updated BIT-1254:
-----------------------------
    Resolution: Won't Fix
        Status: Closed  (was: Open)

> file analysis framework sometimes returns hashes despite missing packets
> ------------------------------------------------------------------------
>
>                 Key: BIT-1254
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1254
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: sample-3streams-hole.pcap
>
>
> Putting the attached sample (3 streams, each with missing packets) though the file analysis framework, in files.log I see hashes for one streams but not the other 2. Should I get any hashes if there are missing packets?
> bro -r sample-3streams-hole.pcap frameworks/files/hash-all-files.bro



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 18 13:15:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Thu, 18 Sep 2014 15:15:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1255) TCP reassembly issue
In-Reply-To: <JIRA.16003.1411050122000@Atlassian.JIRA>
References: <JIRA.16003.1411050122000@Atlassian.JIRA>
	<JIRA.16003.1411050122055@bro-tracker.atlassian.net>
Message-ID: <JIRA.16003.1411050122000.40.1411071307628@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111#comment-18111 ] 

Jimmy Jones commented on BIT-1255:
----------------------------------

I shall pay closer attention to weird.log in the future! Will tweak my production system, thanks.

Based on very little understanding of how Bro's tcp reassembly works... instead of giving up, would it be preferable to signal a gap and carry on processing the rest of the connection?

> TCP reassembly issue
> --------------------
>
>                 Key: BIT-1255
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1255
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: out.pcap
>
>
> Been testing bro with some messy (but valid) TCP streams, using docker and netem (happy to upload a gist if people are interested).
> The attached file reassembles correctly in wireshark, but bro only gives the first 4069 bytes when extracted with the file analysis framework, and obviously the wrong hash (md5 is the URI).



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 18 15:25:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Thu, 18 Sep 2014 17:25:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1255) TCP reassembly issue
In-Reply-To: <JIRA.16003.1411050122000@Atlassian.JIRA>
References: <JIRA.16003.1411050122000@Atlassian.JIRA>
	<JIRA.16003.1411050122055@bro-tracker.atlassian.net>
Message-ID: <JIRA.16003.1411050122000.42.1411079107724@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18112#comment-18112 ] 

Jon Siwek commented on BIT-1255:
--------------------------------

{quote}
Based on very little understanding of how Bro's tcp reassembly works... instead of giving up, would it be preferable to signal a gap and carry on processing the rest of the connection?
{quote}

Good question, not entirely sure the original motivation for that failure mode.  I think in the absence of ACKs and while still trying to impose limits on the size of the reassembly buffer, it may become rather arbitrary which segments get delivered and which get missed, so maybe giving up processing the rest is to avoid a messy/noisy analysis.

> TCP reassembly issue
> --------------------
>
>                 Key: BIT-1255
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1255
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: out.pcap
>
>
> Been testing bro with some messy (but valid) TCP streams, using docker and netem (happy to upload a gist if people are interested).
> The attached file reassembles correctly in wireshark, but bro only gives the first 4069 bytes when extracted with the file analysis framework, and obviously the wrong hash (md5 is the URI).



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-008#64003)

From noreply at bro.org  Fri Sep 19 00:00:26 2014
From: noreply at bro.org (Merge Tracker)
Date: Fri, 19 Sep 2014 00:00:26 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409190700.s8J70Qkd032284@bro-ids.icir.org>


Open Merge Requests
===================

ID            Component    Reporter    Assignee    Updated       For Version  Priority    Summary
------------  -----------  ----------  ----------  ----------  -------------  ----------  -------------------------------
BIT-1252 [1]  Bro          Jon Siwek   -           2014-09-16            2.4  Normal      topic/jsiwek/missing-plugin [2]


Open GitHub Pull Requests
=========================

Issue    Component      User                               Updated     Title
-------  -------------  ---------------------------------  ----------  ------------------------------
#15 [3]  bro            drsirmrpresidentfathercharles [4]  2014-09-18  Rename README to README.md [5]
#1 [6]   cmake          dcode [7]                          2014-09-17  Added RPM metadata [8]
#2 [9]   packet-bricks  asifjamshed [10]                   2014-09-13  typo [11]
#1 [12]  packet-bricks  asifjamshed [13]                   2014-09-13  Fixing minor typos [14]


[1]   BIT-1252                       https://bro-tracker.atlassian.net/browse/BIT-1252
[2]   missing-plugin                 https://github.com/bro/bro/tree/topic/jsiwek/missing-plugin
[3]   Pull Request #15               https://github.com/bro/bro/pull/15
[4]   drsirmrpresidentfathercharles  https://github.com/drsirmrpresidentfathercharles
[5]   Merge Pull Request #15 with    git pull --no-ff --no-commit https://github.com/drsirmrpresidentfathercharles/bro.git master
[6]   Pull Request #1                https://github.com/bro/cmake/pull/1
[7]   dcode                          https://github.com/dcode
[8]   Merge Pull Request #1 with     git pull --no-ff --no-commit https://github.com/dcode/cmake.git patch-1
[9]   Pull Request #2                https://github.com/bro/packet-bricks/pull/2
[10]  asifjamshed                    https://github.com/asifjamshed
[11]  Merge Pull Request #2 with     git pull --no-ff --no-commit https://github.com/asifjamshed/packet-bricks.git master
[12]  Pull Request #1                https://github.com/bro/packet-bricks/pull/1
[13]  asifjamshed                    https://github.com/asifjamshed
[14]  Merge Pull Request #1 with     git pull --no-ff --no-commit https://github.com/asifjamshed/packet-bricks.git patch-1


From noreply at bro.org  Sat Sep 20 00:00:18 2014
From: noreply at bro.org (Merge Tracker)
Date: Sat, 20 Sep 2014 00:00:18 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409200700.s8K70IXd027968@bro-ids.icir.org>


Open Merge Requests
===================

ID            Component    Reporter    Assignee    Updated       For Version  Priority    Summary
------------  -----------  ----------  ----------  ----------  -------------  ----------  -------------------------------
BIT-1252 [1]  Bro          Jon Siwek   -           2014-09-16            2.4  Normal      topic/jsiwek/missing-plugin [2]


Open GitHub Pull Requests
=========================

Issue    Component      User                               Updated     Title
-------  -------------  ---------------------------------  ----------  ------------------------------
#15 [3]  bro            drsirmrpresidentfathercharles [4]  2014-09-18  Rename README to README.md [5]
#1 [6]   cmake          dcode [7]                          2014-09-19  Added RPM metadata [8]
#2 [9]   packet-bricks  asifjamshed [10]                   2014-09-13  typo [11]
#1 [12]  packet-bricks  asifjamshed [13]                   2014-09-13  Fixing minor typos [14]


[1]   BIT-1252                       https://bro-tracker.atlassian.net/browse/BIT-1252
[2]   missing-plugin                 https://github.com/bro/bro/tree/topic/jsiwek/missing-plugin
[3]   Pull Request #15               https://github.com/bro/bro/pull/15
[4]   drsirmrpresidentfathercharles  https://github.com/drsirmrpresidentfathercharles
[5]   Merge Pull Request #15 with    git pull --no-ff --no-commit https://github.com/drsirmrpresidentfathercharles/bro.git master
[6]   Pull Request #1                https://github.com/bro/cmake/pull/1
[7]   dcode                          https://github.com/dcode
[8]   Merge Pull Request #1 with     git pull --no-ff --no-commit https://github.com/dcode/cmake.git patch-1
[9]   Pull Request #2                https://github.com/bro/packet-bricks/pull/2
[10]  asifjamshed                    https://github.com/asifjamshed
[11]  Merge Pull Request #2 with     git pull --no-ff --no-commit https://github.com/asifjamshed/packet-bricks.git master
[12]  Pull Request #1                https://github.com/bro/packet-bricks/pull/1
[13]  asifjamshed                    https://github.com/asifjamshed
[14]  Merge Pull Request #1 with     git pull --no-ff --no-commit https://github.com/asifjamshed/packet-bricks.git patch-1


From noreply at bro.org  Sun Sep 21 00:00:29 2014
From: noreply at bro.org (Merge Tracker)
Date: Sun, 21 Sep 2014 00:00:29 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409210700.s8L70TPQ019639@bro-ids.icir.org>


Open Merge Requests
===================

ID            Component    Reporter    Assignee    Updated       For Version  Priority    Summary
------------  -----------  ----------  ----------  ----------  -------------  ----------  -------------------------------
BIT-1252 [1]  Bro          Jon Siwek   -           2014-09-16            2.4  Normal      topic/jsiwek/missing-plugin [2]


Open GitHub Pull Requests
=========================

Issue    Component      User                               Updated     Title
-------  -------------  ---------------------------------  ----------  ------------------------------
#15 [3]  bro            drsirmrpresidentfathercharles [4]  2014-09-18  Rename README to README.md [5]
#1 [6]   cmake          dcode [7]                          2014-09-19  Added RPM metadata [8]
#2 [9]   packet-bricks  asifjamshed [10]                   2014-09-13  typo [11]
#1 [12]  packet-bricks  asifjamshed [13]                   2014-09-13  Fixing minor typos [14]


[1]   BIT-1252                       https://bro-tracker.atlassian.net/browse/BIT-1252
[2]   missing-plugin                 https://github.com/bro/bro/tree/topic/jsiwek/missing-plugin
[3]   Pull Request #15               https://github.com/bro/bro/pull/15
[4]   drsirmrpresidentfathercharles  https://github.com/drsirmrpresidentfathercharles
[5]   Merge Pull Request #15 with    git pull --no-ff --no-commit https://github.com/drsirmrpresidentfathercharles/bro.git master
[6]   Pull Request #1                https://github.com/bro/cmake/pull/1
[7]   dcode                          https://github.com/dcode
[8]   Merge Pull Request #1 with     git pull --no-ff --no-commit https://github.com/dcode/cmake.git patch-1
[9]   Pull Request #2                https://github.com/bro/packet-bricks/pull/2
[10]  asifjamshed                    https://github.com/asifjamshed
[11]  Merge Pull Request #2 with     git pull --no-ff --no-commit https://github.com/asifjamshed/packet-bricks.git master
[12]  Pull Request #1                https://github.com/bro/packet-bricks/pull/1
[13]  asifjamshed                    https://github.com/asifjamshed
[14]  Merge Pull Request #1 with     git pull --no-ff --no-commit https://github.com/asifjamshed/packet-bricks.git patch-1


From noreply at bro.org  Mon Sep 22 00:00:17 2014
From: noreply at bro.org (Merge Tracker)
Date: Mon, 22 Sep 2014 00:00:17 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409220700.s8M70H98002901@bro-ids.icir.org>


Open Merge Requests
===================

ID            Component    Reporter    Assignee    Updated       For Version  Priority    Summary
------------  -----------  ----------  ----------  ----------  -------------  ----------  -------------------------------
BIT-1252 [1]  Bro          Jon Siwek   -           2014-09-16            2.4  Normal      topic/jsiwek/missing-plugin [2]


Open GitHub Pull Requests
=========================

Issue    Component      User                               Updated     Title
-------  -------------  ---------------------------------  ----------  ------------------------------
#15 [3]  bro            drsirmrpresidentfathercharles [4]  2014-09-18  Rename README to README.md [5]
#1 [6]   cmake          dcode [7]                          2014-09-19  Added RPM metadata [8]
#2 [9]   packet-bricks  asifjamshed [10]                   2014-09-13  typo [11]
#1 [12]  packet-bricks  asifjamshed [13]                   2014-09-13  Fixing minor typos [14]


[1]   BIT-1252                       https://bro-tracker.atlassian.net/browse/BIT-1252
[2]   missing-plugin                 https://github.com/bro/bro/tree/topic/jsiwek/missing-plugin
[3]   Pull Request #15               https://github.com/bro/bro/pull/15
[4]   drsirmrpresidentfathercharles  https://github.com/drsirmrpresidentfathercharles
[5]   Merge Pull Request #15 with    git pull --no-ff --no-commit https://github.com/drsirmrpresidentfathercharles/bro.git master
[6]   Pull Request #1                https://github.com/bro/cmake/pull/1
[7]   dcode                          https://github.com/dcode
[8]   Merge Pull Request #1 with     git pull --no-ff --no-commit https://github.com/dcode/cmake.git patch-1
[9]   Pull Request #2                https://github.com/bro/packet-bricks/pull/2
[10]  asifjamshed                    https://github.com/asifjamshed
[11]  Merge Pull Request #2 with     git pull --no-ff --no-commit https://github.com/asifjamshed/packet-bricks.git master
[12]  Pull Request #1                https://github.com/bro/packet-bricks/pull/1
[13]  asifjamshed                    https://github.com/asifjamshed
[14]  Merge Pull Request #1 with     git pull --no-ff --no-commit https://github.com/asifjamshed/packet-bricks.git patch-1


From jira at bro-tracker.atlassian.net  Mon Sep 22 02:15:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Mon, 22 Sep 2014 04:15:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1257) Same file id generated for potentially
 different files
In-Reply-To: <JIRA.16100.1411377279000@Atlassian.JIRA>
References: <JIRA.16100.1411377279000@Atlassian.JIRA>
	<JIRA.16100.1411377279508@bro-tracker.atlassian.net>
Message-ID: <JIRA.16100.1411377279000.1.1411377307691@Atlassian.JIRA>

Jimmy Jones created BIT-1257:
--------------------------------

             Summary: Same file id generated for potentially different files
                 Key: BIT-1257
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1257
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: 2.3, git/master
         Environment: CentOS 6
            Reporter: Jimmy Jones
         Attachments: fa.bro, sample-samefileid.pcap

Attached sample contains two HTTP downloads of the same URL from the same client, but there are no guarantees that the files is actually the same (no Etags etc - in this case it actually is the same, but lets pretend they were different...). However the file analysis framework seems to give the same file ID in file_name and file_chunk for both downloads.

Think this is something to do with Range requests as doesn't happen if do "normal" HTTP requests.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 22 02:35:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Mon, 22 Sep 2014 04:35:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1258) POP3 question
In-Reply-To: <JIRA.16101.1411378479000@Atlassian.JIRA>
References: <JIRA.16101.1411378479000@Atlassian.JIRA>
	<JIRA.16101.1411378479884@bro-tracker.atlassian.net>
Message-ID: <JIRA.16101.1411378479000.2.1411378507626@Atlassian.JIRA>

Jimmy Jones created BIT-1258:
--------------------------------

             Summary: POP3 question
                 Key: BIT-1258
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1258
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: 2.3
         Environment: CentOS 6
            Reporter: Jimmy Jones
         Attachments: pop.pcap

Using attached POP3 sample (extracted from http://leonward.wordpress.com/2009/04/10/openpacketorg-examplecom-pcap-files/) I get pop3 detected in conn.log but nothing else. Should I get a pop3.log or similar out of the box?



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 22 04:03:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Mon, 22 Sep 2014 06:03:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1259) Change in behaviour of
	connection$resp$size
In-Reply-To: <JIRA.16102.1411383746000@Atlassian.JIRA>
References: <JIRA.16102.1411383746000@Atlassian.JIRA>
	<JIRA.16102.1411383746456@bro-tracker.atlassian.net>
Message-ID: <JIRA.16102.1411383746000.3.1411383787642@Atlassian.JIRA>

Jimmy Jones created BIT-1259:
--------------------------------

             Summary: Change in behaviour of connection$resp$size
                 Key: BIT-1259
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1259
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: git/master
         Environment: CentOS 6
            Reporter: Jimmy Jones
         Attachments: debug.bro, sample-pipeline8b.pcap

Bro 2.3-136 and 2.3-178 give different values for connection$resp$size, possibly as a result of BIT-1246. From the documentation https://www.bro.org/sphinx-git/scripts/base/init-bare.bro.html#type-endpoint I'd expect it to be unaffected by missing packets, but the later version appears to be.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 22 06:00:07 2014
From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA))
Date: Mon, 22 Sep 2014 08:00:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1258) POP3 question
In-Reply-To: <JIRA.16101.1411378479000@Atlassian.JIRA>
References: <JIRA.16101.1411378479000@Atlassian.JIRA>
	<JIRA.16101.1411378479884@bro-tracker.atlassian.net>
Message-ID: <JIRA.16101.1411378479000.5.1411390807990@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Johanna Amann reassigned BIT-1258:
----------------------------------

    Assignee: Johanna Amann

> POP3 question
> -------------
>
>                 Key: BIT-1258
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1258
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>            Assignee: Johanna Amann
>         Attachments: pop.pcap
>
>
> Using attached POP3 sample (extracted from http://leonward.wordpress.com/2009/04/10/openpacketorg-examplecom-pcap-files/) I get pop3 detected in conn.log but nothing else. Should I get a pop3.log or similar out of the box?



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 22 06:04:08 2014
From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA))
Date: Mon, 22 Sep 2014 08:04:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1258) POP3 question
In-Reply-To: <JIRA.16101.1411378479000@Atlassian.JIRA>
References: <JIRA.16101.1411378479000@Atlassian.JIRA>
	<JIRA.16101.1411378479884@bro-tracker.atlassian.net>
Message-ID: <JIRA.16101.1411378479000.7.1411391048187@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18200#comment-18200 ] 

Johanna Amann commented on BIT-1258:
------------------------------------

At the moment, Bro does not ship with the pop3 analyzer enabled by default (or with any scripts for it). If you want to activate the pop3 analyzer, you would have to write a script that enables the analyzer and handles the different protocol events just like they are loaded by default for http, smtp, ftp.

Due to the fact that plain unencrypted pop3 is used very rarely nowadays (and that the analyzer could need a bit of clean-up) we will probably not add any scripts for that in the near future.

> POP3 question
> -------------
>
>                 Key: BIT-1258
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1258
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>            Assignee: Johanna Amann
>         Attachments: pop.pcap
>
>
> Using attached POP3 sample (extracted from http://leonward.wordpress.com/2009/04/10/openpacketorg-examplecom-pcap-files/) I get pop3 detected in conn.log but nothing else. Should I get a pop3.log or similar out of the box?



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 22 06:04:08 2014
From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA))
Date: Mon, 22 Sep 2014 08:04:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1258) POP3 question
In-Reply-To: <JIRA.16101.1411378479000@Atlassian.JIRA>
References: <JIRA.16101.1411378479000@Atlassian.JIRA>
	<JIRA.16101.1411378479884@bro-tracker.atlassian.net>
Message-ID: <JIRA.16101.1411378479000.9.1411391048332@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Johanna Amann updated BIT-1258:
-------------------------------
    Resolution: Won't Fix
        Status: Closed  (was: Open)

> POP3 question
> -------------
>
>                 Key: BIT-1258
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1258
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>            Assignee: Johanna Amann
>         Attachments: pop.pcap
>
>
> Using attached POP3 sample (extracted from http://leonward.wordpress.com/2009/04/10/openpacketorg-examplecom-pcap-files/) I get pop3 detected in conn.log but nothing else. Should I get a pop3.log or similar out of the box?



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 22 06:06:07 2014
From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA))
Date: Mon, 22 Sep 2014 08:06:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1253) Bro 2.3 - 2.3.1 manager dieing on Bivio
	hardware
In-Reply-To: <JIRA.16001.1410956695000@Atlassian.JIRA>
References: <JIRA.16001.1410956695000@Atlassian.JIRA>
	<JIRA.16001.1410956695631@bro-tracker.atlassian.net>
Message-ID: <JIRA.16001.1410956695000.11.1411391167689@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Johanna Amann updated BIT-1253:
-------------------------------
    Fix Version/s: 2.4

> Bro 2.3 - 2.3.1 manager dieing on Bivio hardware
> ------------------------------------------------
>
>                 Key: BIT-1253
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1253
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: 2.3
>         Environment: Bro 2.3 and Bro 2.3.1 
> bivio hardwareLinux CPU.2.6.31-45 has curl 7.36 gperftools 2.2 flex 2.5.39 bison 3.0.2 libpcap 1.1 swig 2.0.8
>            Reporter: Larry Leviton
>             Fix For: 2.4
>
>
> After starting bro up, the bro manager crashes in less than 60 seconds.
> Thanks for any help you can give.
> Sent stack trace to vendor (at bottom), and here was their response:
> Comment(s):	Hello Larry,
> We have duplicated a crash in our lab setup that seems to be identical to that experienced by you. The code has changed quite a bit from 2.1 to 2.3.1, and we suspect a bug was introduced.
> What is going on, seems to be that a writer thread is being terminated, and the destructor for the Ascii writer is called eventually. However, the destructor code does some checks and finds out that proper cleanup has not been done, so it aborts. This does not seem to be due to any library incompatibility, and looks more like maybe a race condition was introduced.
> Since you knows the Bro developers, can you please ask them to take a look this and get back to us? We think it requires their expertise at this point.
> Thank You,
> Hassan.
>  	
> Bivio Case Information:	
> Bivio Case #:	4566243	
> Date Created:	9/02/2014 08:02 AM PDT	
> Stack trace below:
> GNU gdb (GDB) Fedora (6.8.50.20090302-40.fc11) Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "ppc-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> backtrace
> [New Thread 25501]
> [New Thread 25328]
> [New Thread 25378]
> [New Thread 25379]
> [New Thread 25380]
> [New Thread 25381]
> [New Thread 25382]
> [New Thread 25383]
> [New Thread 25384]
> [New Thread 25385]
> [New Thread 25386]
> [New Thread 25389]
> [New Thread 25442]
> warning: Can't read pathname for load map: Input/output error.
> Missing separate debuginfo for /usr/local/lib/libz.so.1
> Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a2/0a0d1fc0d48c2a303af1417ccc03308b9de04a
> Missing separate debuginfo for /usr/local/lib/libtcmalloc.so.4
> Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/27/eaf56bc64810920d55b9530156c1e8ffbfd43e
> Missing separate debuginfo for /usr/local/lib/libcurl.so.4
> Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a7/9a2cebb4abc156495ec0806b1c18015c8eba01
> Reading symbols from /usr/lib/libpcap.so.1...done.
> Loaded symbols for /usr/lib/libpcap.so.1 Reading symbols from /usr/lib/libssl.so.10...done.
> Loaded symbols for /usr/lib/libssl.so.10 Reading symbols from /usr/lib/libcrypto.so.10...done.
> Loaded symbols for /usr/lib/libcrypto.so.10 Reading symbols from /usr/lib/libbind.so.4...done.
> Loaded symbols for /usr/lib/libbind.so.4 Reading symbols from /usr/local/lib/libz.so.1...done.
> Loaded symbols for /usr/local/lib/libz.so.1 Reading symbols from /usr/local/lib/libtcmalloc.so.4...done.
> Loaded symbols for /usr/local/lib/libtcmalloc.so.4 Reading symbols from /usr/local/lib/libcurl.so.4...done.
> Loaded symbols for /usr/local/lib/libcurl.so.4 Reading symbols from /lib/libpthread.so.0...done.
> Loaded symbols for /lib/libpthread.so.0
> Reading symbols from /lib/libdl.so.2...done.
> Loaded symbols for /lib/libdl.so.2
> Reading symbols from /usr/lib/libstdc++.so.6...done.
> Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from /lib/libm.so.6...done.
> Loaded symbols for /lib/libm.so.6
> Reading symbols from /lib/libgcc_s.so.1...done.
> Loaded symbols for /lib/libgcc_s.so.1
> Reading symbols from /lib/libc.so.6...done.
> Loaded symbols for /lib/libc.so.6
> Reading symbols from /usr/lib/libzcp.so...done.
> Loaded symbols for /usr/lib/libzcp.so
> Reading symbols from /lib/libgssapi_krb5.so.2...done.
> Loaded symbols for /lib/libgssapi_krb5.so.2 Reading symbols from /lib/libkrb5.so.3...done.
> Loaded symbols for /lib/libkrb5.so.3
> Reading symbols from /lib/libcom_err.so.2...done.
> Loaded symbols for /lib/libcom_err.so.2
> Reading symbols from /lib/libk5crypto.so.3...done.
> Loaded symbols for /lib/libk5crypto.so.3 Reading symbols from /lib/libresolv.so.2...done.
> Loaded symbols for /lib/libresolv.so.2
> Reading symbols from /lib/librt.so.1...done.
> Loaded symbols for /lib/librt.so.1
> Reading symbols from /lib/ld.so.1...done.
> Loaded symbols for /lib/ld.so.1
> Reading symbols from /lib/libbvsp.so...done.
> Loaded symbols for /lib/libbvsp.so
> Reading symbols from /lib/libbcon.so...done.
> Loaded symbols for /lib/libbcon.so
> Reading symbols from /lib/libkrb5support.so.0...done.
> Loaded symbols for /lib/libkrb5support.so.0 Reading symbols from /lib/libkeyutils.so.1...done.
> Loaded symbols for /lib/libkeyutils.so.1 Reading symbols from /usr/lib/libxml2.so.2...done.
> Loaded symbols for /usr/lib/libxml2.so.2 Reading symbols from /lib/libhmlibs.so...done.
> Loaded symbols for /lib/libhmlibs.so
> Reading symbols from /lib/libhmolddb.so...done.
> Loaded symbols for /lib/libhmolddb.so
> Reading symbols from /lib/libcf.so...done.
> Loaded symbols for /lib/libcf.so
> Reading symbols from /lib/libbvsep.so...done.
> Loaded symbols for /lib/libbvsep.so
> Reading symbols from /usr/lib/libnrddi.so...done.
> Loaded symbols for /usr/lib/libnrddi.so
> Reading symbols from /lib/libselinux.so.1...done.
> Loaded symbols for /lib/libselinux.so.1
> Core was generated by `/var/tmp/bro/spool/tmp/bro -U .status -p broctl -p broctl-live -p local -p mana'.
> Program terminated with signal 6, Aborted.
> #0  0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> 56	../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> 	in ../nptl/sysdeps/unix/sysv/linux/raise.c
> Missing separate debuginfos, use: debuginfo-install e2fsprogs-libs-1.41.9-2.fc11.ppc glibc-2.17-4.fc11.ppc keyutils-libs-1.2-5.fc11.ppc krb5-libs-1.9.3-1.fc11.ppc libbind-6.0-1.fc11.ppc libgcc-4.4.1-2.fc11.ppc libselinux-2.0.80-1.fc11.ppc libstdc++-4.4.1-2.fc11.ppc libxml2-2.7.6-1.fc11.ppc openssl-libs-1.0.1e-37.fc11.1.ppc
> (gdb) backtrace
> #0  0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1  0x0f6d0de0 in *__GI_abort () at abort.c:90
> #2  0x1024be70 in logging::writer::Ascii::~Ascii (this=0x11a87200, __in_chrg=<value optimized out>)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/logging/writers/Ascii.cc:186
> #3  0x10236b70 in threading::Manager::Process (this=0x10dae180)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/threading/Manager.cc:171
> #4  0x101a5400 in net_run () at /bivio/scsi/b/levitonl/bro-2.3.1/src/Net.cc:389
> #5  0x100f7554 in main (argc=<value optimized out>, argv=<value optimized out>)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/main.cc:1165
> Current language:  auto; currently minimal
> (gdb)
> #0  0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1  0x0f6d0de0 in *__GI_abort () at abort.c:90
> #2  0x1024be70 in logging::writer::Ascii::~Ascii (this=0x11a87200, __in_chrg=<value optimized out>)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/logging/writers/Ascii.cc:186
> #3  0x10236b70 in threading::Manager::Process (this=0x10dae180)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/threading/Manager.cc:171
> #4  0x101a5400 in net_run () at /bivio/scsi/b/levitonl/bro-2.3.1/src/Net.cc:389
> #5  0x100f7554 in main (argc=<value optimized out>, argv=<value optimized out>)
>     at /bivio/scsi/b/levitonl/bro-2.3.1/src/main.cc:1165
> (gdb) quit



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 22 06:55:08 2014
From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA))
Date: Mon, 22 Sep 2014 08:55:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1257) Same file id generated for potentially
 different files
In-Reply-To: <JIRA.16100.1411377279000@Atlassian.JIRA>
References: <JIRA.16100.1411377279000@Atlassian.JIRA>
	<JIRA.16100.1411377279508@bro-tracker.atlassian.net>
Message-ID: <JIRA.16100.1411377279000.13.1411394108129@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18201#comment-18201 ] 

Seth Hall commented on BIT-1257:
--------------------------------

Unfortunately there isn't a single consistent way to identify file chunks over HTTP and rely on those being from the same file over time.

Take a look at scripts/base/protocols/http/files.bro.  If you look at the get_file_handle handle function defined in that file I point you at you can see our default mechanism for choosing file IDs for files over HTTP.  If you want to replace that with your own, take a look at the bottom of  that file.  You can call the Files::register_protocol function with your own get_file_handle equivalent to make Bro behave in some different way.

> Same file id generated for potentially different files
> ------------------------------------------------------
>
>                 Key: BIT-1257
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1257
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: fa.bro, sample-samefileid.pcap
>
>
> Attached sample contains two HTTP downloads of the same URL from the same client, but there are no guarantees that the files is actually the same (no Etags etc - in this case it actually is the same, but lets pretend they were different...). However the file analysis framework seems to give the same file ID in file_name and file_chunk for both downloads.
> Think this is something to do with Range requests as doesn't happen if do "normal" HTTP requests.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 22 08:55:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Mon, 22 Sep 2014 10:55:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1257) Same file id generated for potentially
 different files
In-Reply-To: <JIRA.16100.1411377279000@Atlassian.JIRA>
References: <JIRA.16100.1411377279000@Atlassian.JIRA>
	<JIRA.16100.1411377279508@bro-tracker.atlassian.net>
Message-ID: <JIRA.16100.1411377279000.15.1411401307848@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18202#comment-18202 ] 

Jimmy Jones commented on BIT-1257:
----------------------------------

Agreed there isn't a good way to identify it being the same file. However with the attached sample the two downloads do get merged into one, which I think is incorrect behavior.

> Same file id generated for potentially different files
> ------------------------------------------------------
>
>                 Key: BIT-1257
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1257
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: fa.bro, sample-samefileid.pcap
>
>
> Attached sample contains two HTTP downloads of the same URL from the same client, but there are no guarantees that the files is actually the same (no Etags etc - in this case it actually is the same, but lets pretend they were different...). However the file analysis framework seems to give the same file ID in file_name and file_chunk for both downloads.
> Think this is something to do with Range requests as doesn't happen if do "normal" HTTP requests.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 22 11:02:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Mon, 22 Sep 2014 13:02:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1259) Change in behaviour of
	connection$resp$size
In-Reply-To: <JIRA.16102.1411383746000@Atlassian.JIRA>
References: <JIRA.16102.1411383746000@Atlassian.JIRA>
	<JIRA.16102.1411383746456@bro-tracker.atlassian.net>
Message-ID: <JIRA.16102.1411383746000.17.1411408927812@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1259:
---------------------------
    Fix Version/s: 2.4

> Change in behaviour of connection$resp$size
> -------------------------------------------
>
>                 Key: BIT-1259
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1259
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>             Fix For: 2.4
>
>         Attachments: debug.bro, sample-pipeline8b.pcap
>
>
> Bro 2.3-136 and 2.3-178 give different values for connection$resp$size, possibly as a result of BIT-1246. From the documentation https://www.bro.org/sphinx-git/scripts/base/init-bare.bro.html#type-endpoint I'd expect it to be unaffected by missing packets, but the later version appears to be.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 22 11:05:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Mon, 22 Sep 2014 13:05:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1259) Change in behaviour of
	connection$resp$size
In-Reply-To: <JIRA.16102.1411383746000@Atlassian.JIRA>
References: <JIRA.16102.1411383746000@Atlassian.JIRA>
	<JIRA.16102.1411383746456@bro-tracker.atlassian.net>
Message-ID: <JIRA.16102.1411383746000.20.1411409107933@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1259:
---------------------------
    Resolution: Fixed
        Status: Closed  (was: Open)

> Change in behaviour of connection$resp$size
> -------------------------------------------
>
>                 Key: BIT-1259
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1259
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>             Fix For: 2.4
>
>         Attachments: debug.bro, sample-pipeline8b.pcap
>
>
> Bro 2.3-136 and 2.3-178 give different values for connection$resp$size, possibly as a result of BIT-1246. From the documentation https://www.bro.org/sphinx-git/scripts/base/init-bare.bro.html#type-endpoint I'd expect it to be unaffected by missing packets, but the later version appears to be.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 22 11:05:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Mon, 22 Sep 2014 13:05:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1259) Change in behaviour of
	connection$resp$size
In-Reply-To: <JIRA.16102.1411383746000@Atlassian.JIRA>
References: <JIRA.16102.1411383746000@Atlassian.JIRA>
	<JIRA.16102.1411383746456@bro-tracker.atlassian.net>
Message-ID: <JIRA.16102.1411383746000.19.1411409107841@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18203#comment-18203 ] 

Jon Siwek commented on BIT-1259:
--------------------------------

Yes, the change from BIT-1246 could cause the same data to be delivered twice in some cases, so your sample was triggering http_message_done at an earlier point than it should have (the content-length was being hit sooner due to the duplicate data sent for HTTP analysis).  Should be fixed now in git/master.

> Change in behaviour of connection$resp$size
> -------------------------------------------
>
>                 Key: BIT-1259
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1259
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>             Fix For: 2.4
>
>         Attachments: debug.bro, sample-pipeline8b.pcap
>
>
> Bro 2.3-136 and 2.3-178 give different values for connection$resp$size, possibly as a result of BIT-1246. From the documentation https://www.bro.org/sphinx-git/scripts/base/init-bare.bro.html#type-endpoint I'd expect it to be unaffected by missing packets, but the later version appears to be.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 22 11:17:07 2014
From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA))
Date: Mon, 22 Sep 2014 13:17:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1257) Same file id generated for potentially
 different files
In-Reply-To: <JIRA.16100.1411377279000@Atlassian.JIRA>
References: <JIRA.16100.1411377279000@Atlassian.JIRA>
	<JIRA.16100.1411377279508@bro-tracker.atlassian.net>
Message-ID: <JIRA.16100.1411377279000.23.1411409827701@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18204#comment-18204 ] 

Seth Hall commented on BIT-1257:
--------------------------------

Do you have any suggestions on what our default file identification code should look like for HTTP?  It's something that you have the ability to change without making any core changes in Bro as I pointed out in my previous comment.

> Same file id generated for potentially different files
> ------------------------------------------------------
>
>                 Key: BIT-1257
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1257
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: fa.bro, sample-samefileid.pcap
>
>
> Attached sample contains two HTTP downloads of the same URL from the same client, but there are no guarantees that the files is actually the same (no Etags etc - in this case it actually is the same, but lets pretend they were different...). However the file analysis framework seems to give the same file ID in file_name and file_chunk for both downloads.
> Think this is something to do with Range requests as doesn't happen if do "normal" HTTP requests.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 22 11:45:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Mon, 22 Sep 2014 13:45:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1260) topic/jsiwek/improve_comm_loop
In-Reply-To: <JIRA.16103.1411411475000@Atlassian.JIRA>
References: <JIRA.16103.1411411475000@Atlassian.JIRA>
	<JIRA.16103.1411411475304@bro-tracker.atlassian.net>
Message-ID: <JIRA.16103.1411411475000.24.1411411507630@Atlassian.JIRA>

Jon Siwek created BIT-1260:
------------------------------

             Summary: topic/jsiwek/improve_comm_loop
                 Key: BIT-1260
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1260
             Project: Bro Issue Tracker
          Issue Type: Improvement
          Components: Bro
            Reporter: Jon Siwek
             Fix For: 2.4


This branch removes timeouts from the remote communication loop and so reduces cpu usage during the times it's supposed to be idle.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 22 11:45:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Mon, 22 Sep 2014 13:45:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1260) topic/jsiwek/improve_comm_loop
In-Reply-To: <JIRA.16103.1411411475000@Atlassian.JIRA>
References: <JIRA.16103.1411411475000@Atlassian.JIRA>
	<JIRA.16103.1411411475304@bro-tracker.atlassian.net>
Message-ID: <JIRA.16103.1411411475000.25.1411411507682@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1260:
---------------------------
    Status: Merge Request  (was: Open)

> topic/jsiwek/improve_comm_loop
> ------------------------------
>
>                 Key: BIT-1260
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1260
>             Project: Bro Issue Tracker
>          Issue Type: Improvement
>          Components: Bro
>            Reporter: Jon Siwek
>             Fix For: 2.4
>
>
> This branch removes timeouts from the remote communication loop and so reduces cpu usage during the times it's supposed to be idle.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 22 12:10:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Mon, 22 Sep 2014 14:10:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1257) Same file id generated for potentially
 different files
In-Reply-To: <JIRA.16100.1411377279000@Atlassian.JIRA>
References: <JIRA.16100.1411377279000@Atlassian.JIRA>
	<JIRA.16100.1411377279508@bro-tracker.atlassian.net>
Message-ID: <JIRA.16100.1411377279000.27.1411413007959@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18205#comment-18205 ] 

Jimmy Jones commented on BIT-1257:
----------------------------------

scripts/base/protocols/http/files.bro:
{quote}
Any multipart responses from the server are pieces of same file
that correspond to range requests, so don't use mime depth to
identify the file.
{quote}
Might be true for multipart responses, however this code is also catching completely separate HTTP sessions, where the URL could be dynamically generated content, so would merge different pieces of content into a corrupt output, which I don't think is a safe default.

> Same file id generated for potentially different files
> ------------------------------------------------------
>
>                 Key: BIT-1257
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1257
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: fa.bro, sample-samefileid.pcap
>
>
> Attached sample contains two HTTP downloads of the same URL from the same client, but there are no guarantees that the files is actually the same (no Etags etc - in this case it actually is the same, but lets pretend they were different...). However the file analysis framework seems to give the same file ID in file_name and file_chunk for both downloads.
> Think this is something to do with Range requests as doesn't happen if do "normal" HTTP requests.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 22 13:02:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Mon, 22 Sep 2014 15:02:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1257) Same file id generated for potentially
 different files
In-Reply-To: <JIRA.16100.1411377279000@Atlassian.JIRA>
References: <JIRA.16100.1411377279000@Atlassian.JIRA>
	<JIRA.16100.1411377279508@bro-tracker.atlassian.net>
Message-ID: <JIRA.16100.1411377279000.29.1411416127720@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18205#comment-18205 ] 

Jimmy Jones edited comment on BIT-1257 at 9/22/14 3:01 PM:
-----------------------------------------------------------

scripts/base/protocols/http/files.bro:
{code}
# Any multipart responses from the server are pieces of same file
# that correspond to range requests, so don't use mime depth to
# identify the file.
{code}
Might be true for multipart responses, however this code is also catching completely separate HTTP sessions, where the URL could be dynamically generated content, so would merge different pieces of content into a corrupt output, which I don't think is a safe default.

Possibly should be (same as non-range case, but without mime depth - so is limited to merging a single response):
{code}
return cat(Analyzer::ANALYZER_HTTP, c$start_time, is_orig,
		           c$http$trans_depth, id_string(c$id));
{code}


was (Author: jimmyjones2):
scripts/base/protocols/http/files.bro:
{quote}
Any multipart responses from the server are pieces of same file
that correspond to range requests, so don't use mime depth to
identify the file.
{quote}
Might be true for multipart responses, however this code is also catching completely separate HTTP sessions, where the URL could be dynamically generated content, so would merge different pieces of content into a corrupt output, which I don't think is a safe default.

> Same file id generated for potentially different files
> ------------------------------------------------------
>
>                 Key: BIT-1257
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1257
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: fa.bro, sample-samefileid.pcap
>
>
> Attached sample contains two HTTP downloads of the same URL from the same client, but there are no guarantees that the files is actually the same (no Etags etc - in this case it actually is the same, but lets pretend they were different...). However the file analysis framework seems to give the same file ID in file_name and file_chunk for both downloads.
> Think this is something to do with Range requests as doesn't happen if do "normal" HTTP requests.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From noreply at bro.org  Tue Sep 23 00:00:58 2014
From: noreply at bro.org (Merge Tracker)
Date: Tue, 23 Sep 2014 00:00:58 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409230700.s8N70wL1010584@bro-ids.icir.org>


Open Merge Requests
===================

ID            Component    Reporter    Assignee    Updated       For Version  Priority    Summary
------------  -----------  ----------  ----------  ----------  -------------  ----------  ----------------------------------
BIT-1260 [1]  Bro          Jon Siwek   -           2014-09-22            2.4  Normal      topic/jsiwek/improve_comm_loop [2]
BIT-1252 [3]  Bro          Jon Siwek   -           2014-09-16            2.4  Normal      topic/jsiwek/missing-plugin [4]


Open GitHub Pull Requests
=========================

Issue    Component    User                               Updated     Title
-------  -----------  ---------------------------------  ----------  ------------------------------
#15 [5]  bro          drsirmrpresidentfathercharles [6]  2014-09-18  Rename README to README.md [7]


[1]  BIT-1260                       https://bro-tracker.atlassian.net/browse/BIT-1260
[2]  improve_comm_loop              https://github.com/bro/bro/tree/topic/jsiwek/improve_comm_loop
[3]  BIT-1252                       https://bro-tracker.atlassian.net/browse/BIT-1252
[4]  missing-plugin                 https://github.com/bro/bro/tree/topic/jsiwek/missing-plugin
[5]  Pull Request #15               https://github.com/bro/bro/pull/15
[6]  drsirmrpresidentfathercharles  https://github.com/drsirmrpresidentfathercharles
[7]  Merge Pull Request #15 with    git pull --no-ff --no-commit https://github.com/drsirmrpresidentfathercharles/bro.git master


From jira at bro-tracker.atlassian.net  Tue Sep 23 19:37:07 2014
From: jira at bro-tracker.atlassian.net (AK (JIRA))
Date: Tue, 23 Sep 2014 21:37:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1261) Bro plugin documentation incomplete
In-Reply-To: <JIRA.16104.1411526219000@Atlassian.JIRA>
References: <JIRA.16104.1411526219000@Atlassian.JIRA>
	<JIRA.16104.1411526219747@bro-tracker.atlassian.net>
Message-ID: <JIRA.16104.1411526219000.30.1411526227645@Atlassian.JIRA>

AK created BIT-1261:
-----------------------

             Summary: Bro plugin documentation incomplete
                 Key: BIT-1261
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1261
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: bro-aux
            Reporter: AK


I found some issues with the steps found here: <https://www.bro.org/sphinx-git/devel/plugins.html>. The three important changes include running configure with the Bro directory option set, the name of the *.bif file matches the plugin name specified to init-plugin, and the call to byte_vec() in the return line of the example rot13 function. Other than that, everything worked rather nicely.

The steps should be:
cd aux/bro-aux/plugin-support
mkdir rot13-plugin
cd rot13-plugin
../init-plugin Demo Rot13
# edit src/rot13.bif to include the following, note the difference in the return line

module CaesarCipher;
function rot13%(s: string%) : string
    %{
    char* rot13 = copy_string(s->CheckString());

    for ( char* p = rot13; *p; p++ )
        {
        char b = islower(*p) ? 'a' : 'A';
        *p  = (*p - b + 13) % 26 + b;
        }

    return new StringVal(new BroString(1, byte_vec(rot13), strlen(rot13)));
    %}

./configure --bro-dist=%BRO_DIR%
make
export BRO_PLUGIN_PATH=$BRO_PLUGIN_PATH:%BRO_DIR%/bro/aux/bro-aux/plugin-support/MY_PLUGIN_NAME
%BRO_DIR%/build/src/bro -N
%BRO_DIR%/bro -e 'print CaesarCipher::rot13("Hello")'





--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From noreply at bro.org  Wed Sep 24 00:00:24 2014
From: noreply at bro.org (Merge Tracker)
Date: Wed, 24 Sep 2014 00:00:24 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409240700.s8O70OgU012739@bro-ids.icir.org>


Open Merge Requests
===================

ID            Component    Reporter    Assignee    Updated       For Version  Priority    Summary
------------  -----------  ----------  ----------  ----------  -------------  ----------  ----------------------------------
BIT-1260 [1]  Bro          Jon Siwek   -           2014-09-22            2.4  Normal      topic/jsiwek/improve_comm_loop [2]
BIT-1252 [3]  Bro          Jon Siwek   -           2014-09-16            2.4  Normal      topic/jsiwek/missing-plugin [4]


Open Fastpath Commits
======================

Commit       Component    Author     Date        Summary
-----------  -----------  ---------  ----------  ----------------------------------------------------------
8d283db [5]  bro          Seth Hall  2014-09-23  Adds a "node" field to Intel::Seen struture and intel.log.


Open GitHub Pull Requests
=========================

Issue    Component    User                               Updated     Title
-------  -----------  ---------------------------------  ----------  ------------------------------
#15 [6]  bro          drsirmrpresidentfathercharles [7]  2014-09-18  Rename README to README.md [8]


[1]  BIT-1260                       https://bro-tracker.atlassian.net/browse/BIT-1260
[2]  improve_comm_loop              https://github.com/bro/bro/tree/topic/jsiwek/improve_comm_loop
[3]  BIT-1252                       https://bro-tracker.atlassian.net/browse/BIT-1252
[4]  missing-plugin                 https://github.com/bro/bro/tree/topic/jsiwek/missing-plugin
[5]  8d283db                        https://github.com/bro/bro/commit/8d283db63b6e433dbd3f4775447905a9f6b43326
[6]  Pull Request #15               https://github.com/bro/bro/pull/15
[7]  drsirmrpresidentfathercharles  https://github.com/drsirmrpresidentfathercharles
[8]  Merge Pull Request #15 with    git pull --no-ff --no-commit https://github.com/drsirmrpresidentfathercharles/bro.git master


From jira at bro-tracker.atlassian.net  Wed Sep 24 07:46:08 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Wed, 24 Sep 2014 09:46:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1261) Bro plugin documentation incomplete
In-Reply-To: <JIRA.16104.1411526219000@Atlassian.JIRA>
References: <JIRA.16104.1411526219000@Atlassian.JIRA>
	<JIRA.16104.1411526219747@bro-tracker.atlassian.net>
Message-ID: <JIRA.16104.1411526219000.32.1411569968510@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1261:
---------------------------
    Assignee: Jon Siwek

> Bro plugin documentation incomplete
> -----------------------------------
>
>                 Key: BIT-1261
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1261
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: bro-aux
>            Reporter: AK
>            Assignee: Jon Siwek
>              Labels: documentation
>
> I found some issues with the steps found here: <https://www.bro.org/sphinx-git/devel/plugins.html>. The three important changes include running configure with the Bro directory option set, the name of the *.bif file matches the plugin name specified to init-plugin, and the call to byte_vec() in the return line of the example rot13 function. Other than that, everything worked rather nicely.
> The steps should be:
> cd aux/bro-aux/plugin-support
> mkdir rot13-plugin
> cd rot13-plugin
> ../init-plugin Demo Rot13
> # edit src/rot13.bif to include the following, note the difference in the return line
> module CaesarCipher;
> function rot13%(s: string%) : string
>     %{
>     char* rot13 = copy_string(s->CheckString());
>     for ( char* p = rot13; *p; p++ )
>         {
>         char b = islower(*p) ? 'a' : 'A';
>         *p  = (*p - b + 13) % 26 + b;
>         }
>     return new StringVal(new BroString(1, byte_vec(rot13), strlen(rot13)));
>     %}
> ./configure --bro-dist=%BRO_DIR%
> make
> export BRO_PLUGIN_PATH=$BRO_PLUGIN_PATH:%BRO_DIR%/bro/aux/bro-aux/plugin-support/MY_PLUGIN_NAME
> %BRO_DIR%/build/src/bro -N
> %BRO_DIR%/bro -e 'print CaesarCipher::rot13("Hello")'



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Wed Sep 24 08:08:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Wed, 24 Sep 2014 10:08:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1261) Bro plugin documentation incomplete
In-Reply-To: <JIRA.16104.1411526219000@Atlassian.JIRA>
References: <JIRA.16104.1411526219000@Atlassian.JIRA>
	<JIRA.16104.1411526219747@bro-tracker.atlassian.net>
Message-ID: <JIRA.16104.1411526219000.34.1411571287684@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1261:
---------------------------
    Fix Version/s: 2.4

> Bro plugin documentation incomplete
> -----------------------------------
>
>                 Key: BIT-1261
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1261
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: bro-aux
>            Reporter: AK
>            Assignee: Jon Siwek
>              Labels: documentation
>             Fix For: 2.4
>
>
> I found some issues with the steps found here: <https://www.bro.org/sphinx-git/devel/plugins.html>. The three important changes include running configure with the Bro directory option set, the name of the *.bif file matches the plugin name specified to init-plugin, and the call to byte_vec() in the return line of the example rot13 function. Other than that, everything worked rather nicely.
> The steps should be:
> cd aux/bro-aux/plugin-support
> mkdir rot13-plugin
> cd rot13-plugin
> ../init-plugin Demo Rot13
> # edit src/rot13.bif to include the following, note the difference in the return line
> module CaesarCipher;
> function rot13%(s: string%) : string
>     %{
>     char* rot13 = copy_string(s->CheckString());
>     for ( char* p = rot13; *p; p++ )
>         {
>         char b = islower(*p) ? 'a' : 'A';
>         *p  = (*p - b + 13) % 26 + b;
>         }
>     return new StringVal(new BroString(1, byte_vec(rot13), strlen(rot13)));
>     %}
> ./configure --bro-dist=%BRO_DIR%
> make
> export BRO_PLUGIN_PATH=$BRO_PLUGIN_PATH:%BRO_DIR%/bro/aux/bro-aux/plugin-support/MY_PLUGIN_NAME
> %BRO_DIR%/build/src/bro -N
> %BRO_DIR%/bro -e 'print CaesarCipher::rot13("Hello")'



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Wed Sep 24 08:19:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Wed, 24 Sep 2014 10:19:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1261) Bro plugin documentation incomplete
In-Reply-To: <JIRA.16104.1411526219000@Atlassian.JIRA>
References: <JIRA.16104.1411526219000@Atlassian.JIRA>
	<JIRA.16104.1411526219747@bro-tracker.atlassian.net>
Message-ID: <JIRA.16104.1411526219000.36.1411571947684@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jon Siwek updated BIT-1261:
---------------------------
    Resolution: Fixed
        Status: Closed  (was: Open)

> Bro plugin documentation incomplete
> -----------------------------------
>
>                 Key: BIT-1261
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1261
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: bro-aux
>            Reporter: AK
>            Assignee: Jon Siwek
>              Labels: documentation
>             Fix For: 2.4
>
>
> I found some issues with the steps found here: <https://www.bro.org/sphinx-git/devel/plugins.html>. The three important changes include running configure with the Bro directory option set, the name of the *.bif file matches the plugin name specified to init-plugin, and the call to byte_vec() in the return line of the example rot13 function. Other than that, everything worked rather nicely.
> The steps should be:
> cd aux/bro-aux/plugin-support
> mkdir rot13-plugin
> cd rot13-plugin
> ../init-plugin Demo Rot13
> # edit src/rot13.bif to include the following, note the difference in the return line
> module CaesarCipher;
> function rot13%(s: string%) : string
>     %{
>     char* rot13 = copy_string(s->CheckString());
>     for ( char* p = rot13; *p; p++ )
>         {
>         char b = islower(*p) ? 'a' : 'A';
>         *p  = (*p - b + 13) % 26 + b;
>         }
>     return new StringVal(new BroString(1, byte_vec(rot13), strlen(rot13)));
>     %}
> ./configure --bro-dist=%BRO_DIR%
> make
> export BRO_PLUGIN_PATH=$BRO_PLUGIN_PATH:%BRO_DIR%/bro/aux/bro-aux/plugin-support/MY_PLUGIN_NAME
> %BRO_DIR%/build/src/bro -N
> %BRO_DIR%/bro -e 'print CaesarCipher::rot13("Hello")'



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Wed Sep 24 23:09:07 2014
From: jira at bro-tracker.atlassian.net (Alexander Zatserkovnyy (JIRA))
Date: Thu, 25 Sep 2014 01:09:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1262) workers fail to start with dnacluster (
 vers. > 2.3-172 )
In-Reply-To: <JIRA.16105.1411625340000@Atlassian.JIRA>
References: <JIRA.16105.1411625340000@Atlassian.JIRA>
	<JIRA.16105.1411625340170@bro-tracker.atlassian.net>
Message-ID: <JIRA.16105.1411625340000.37.1411625347646@Atlassian.JIRA>

Alexander Zatserkovnyy created BIT-1262:
-------------------------------------------

             Summary: workers fail to start with dnacluster ( vers. > 2.3-172 )
                 Key: BIT-1262
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1262
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: 2.3
         Environment: ./pfdnacluster_master -d -i dna0 -c 10 -n 4

node.cfg:
[worker-1]
type=worker
host=localhost
interface=dnacluster:10
#interface=dna0
lb_method=pf_ring
lb_procs=4

            Reporter: Alexander Zatserkovnyy


The configuration works well in 2.3.1 . 

After upgrade up to 2.3-183, workers fail with the message:
"fatal error: type of packet source 'dnacluster' not recognized, or mode not supported" in the stderr .



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From noreply at bro.org  Thu Sep 25 00:00:17 2014
From: noreply at bro.org (Merge Tracker)
Date: Thu, 25 Sep 2014 00:00:17 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409250700.s8P70H7v021434@bro-ids.icir.org>


Open Merge Requests
===================

ID            Component    Reporter    Assignee    Updated       For Version  Priority    Summary
------------  -----------  ----------  ----------  ----------  -------------  ----------  ----------------------------------
BIT-1260 [1]  Bro          Jon Siwek   -           2014-09-22            2.4  Normal      topic/jsiwek/improve_comm_loop [2]
BIT-1252 [3]  Bro          Jon Siwek   -           2014-09-16            2.4  Normal      topic/jsiwek/missing-plugin [4]


Open GitHub Pull Requests
=========================

Issue    Component    User                               Updated     Title
-------  -----------  ---------------------------------  ----------  ------------------------------
#15 [5]  bro          drsirmrpresidentfathercharles [6]  2014-09-18  Rename README to README.md [7]


[1]  BIT-1260                       https://bro-tracker.atlassian.net/browse/BIT-1260
[2]  improve_comm_loop              https://github.com/bro/bro/tree/topic/jsiwek/improve_comm_loop
[3]  BIT-1252                       https://bro-tracker.atlassian.net/browse/BIT-1252
[4]  missing-plugin                 https://github.com/bro/bro/tree/topic/jsiwek/missing-plugin
[5]  Pull Request #15               https://github.com/bro/bro/pull/15
[6]  drsirmrpresidentfathercharles  https://github.com/drsirmrpresidentfathercharles
[7]  Merge Pull Request #15 with    git pull --no-ff --no-commit https://github.com/drsirmrpresidentfathercharles/bro.git master


From jira at bro-tracker.atlassian.net  Thu Sep 25 05:34:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Thu, 25 Sep 2014 07:34:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1257) Same file id generated for potentially
 different files
In-Reply-To: <JIRA.16100.1411377279000@Atlassian.JIRA>
References: <JIRA.16100.1411377279000@Atlassian.JIRA>
	<JIRA.16100.1411377279508@bro-tracker.atlassian.net>
Message-ID: <JIRA.16100.1411377279000.39.1411648447712@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18206#comment-18206 ] 

Jimmy Jones commented on BIT-1257:
----------------------------------

Tested above with testing/btest/Traces/http/byteranges.pcap and produces same output as before with ANALYZER_EXTRACT (one file with all the chunks), which I presume was the intention of the code, and tested with my sample which now produces 2 files, as expected. Could this change be merged please?

> Same file id generated for potentially different files
> ------------------------------------------------------
>
>                 Key: BIT-1257
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1257
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: fa.bro, sample-samefileid.pcap
>
>
> Attached sample contains two HTTP downloads of the same URL from the same client, but there are no guarantees that the files is actually the same (no Etags etc - in this case it actually is the same, but lets pretend they were different...). However the file analysis framework seems to give the same file ID in file_name and file_chunk for both downloads.
> Think this is something to do with Range requests as doesn't happen if do "normal" HTTP requests.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 25 06:14:08 2014
From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA))
Date: Thu, 25 Sep 2014 08:14:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1257) Same file id generated for potentially
 different files
In-Reply-To: <JIRA.16100.1411377279000@Atlassian.JIRA>
References: <JIRA.16100.1411377279000@Atlassian.JIRA>
	<JIRA.16100.1411377279508@bro-tracker.atlassian.net>
Message-ID: <JIRA.16100.1411377279000.41.1411650848460@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18207#comment-18207 ] 

Seth Hall commented on BIT-1257:
--------------------------------

I suspect your changes break some of our tests.  Have you run with your changes over the full test suite?

> Same file id generated for potentially different files
> ------------------------------------------------------
>
>                 Key: BIT-1257
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1257
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: fa.bro, sample-samefileid.pcap
>
>
> Attached sample contains two HTTP downloads of the same URL from the same client, but there are no guarantees that the files is actually the same (no Etags etc - in this case it actually is the same, but lets pretend they were different...). However the file analysis framework seems to give the same file ID in file_name and file_chunk for both downloads.
> Think this is something to do with Range requests as doesn't happen if do "normal" HTTP requests.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 25 08:06:07 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Thu, 25 Sep 2014 10:06:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1262) workers fail to start with dnacluster (
 vers. > 2.3-172 )
In-Reply-To: <JIRA.16105.1411625340000@Atlassian.JIRA>
References: <JIRA.16105.1411625340000@Atlassian.JIRA>
	<JIRA.16105.1411625340170@bro-tracker.atlassian.net>
Message-ID: <JIRA.16105.1411625340000.43.1411657567710@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robin Sommer updated BIT-1262:
------------------------------
    Resolution: Duplicate
        Status: Closed  (was: Open)

I suspect this is the same problem as in BIT-1249. Closing as a duplicate, fix will be tracked there.

> workers fail to start with dnacluster ( vers. > 2.3-172 )
> ---------------------------------------------------------
>
>                 Key: BIT-1262
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1262
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: 2.3
>         Environment: ./pfdnacluster_master -d -i dna0 -c 10 -n 4
> node.cfg:
> [worker-1]
> type=worker
> host=localhost
> interface=dnacluster:10
> #interface=dna0
> lb_method=pf_ring
> lb_procs=4
>            Reporter: Alexander Zatserkovnyy
>
> The configuration works well in 2.3.1 . 
> After upgrade up to 2.3-183, workers fail with the message:
> "fatal error: type of packet source 'dnacluster' not recognized, or mode not supported" in the stderr .



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 25 10:45:07 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Thu, 25 Sep 2014 12:45:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1252) topic/jsiwek/missing-plugin
In-Reply-To: <JIRA.16000.1410880686000@Atlassian.JIRA>
References: <JIRA.16000.1410880686000@Atlassian.JIRA>
	<JIRA.16000.1410880686994@bro-tracker.atlassian.net>
Message-ID: <JIRA.16000.1410880686000.45.1411667107953@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robin Sommer reassigned BIT-1252:
---------------------------------

    Assignee: Robin Sommer

> topic/jsiwek/missing-plugin
> ---------------------------
>
>                 Key: BIT-1252
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1252
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>            Reporter: Jon Siwek
>            Assignee: Robin Sommer
>             Fix For: 2.4
>
>
> Trying to activate a non-existing dynamic plugin gave a hard to understand error message and core dumps:
> {noformat}
> $ bro Bro::NOPE
> error in /Users/jsiwek/Projects/bro/bro/scripts/base/init-bare.bro, line 1: plugin Bro::NOPE is not available
> internal error in /Users/jsiwek/Projects/bro/bro/build/scripts/base/bif/const.bif.bro, line 33: internal variable state_dir missing
> Abort trap: 6
> {noformat}
> It also seems that {{bro -N Bro::NOPE}}, which is used by the {{testing/scripts/has-writer}} helper script, is interpreted as a request to activate a dynamic plugin instead of just query if a plugin exists w/ that name.  Not sure the intention: I just changed the helper script to grep the output of {{bro -N}} instead.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 25 12:50:07 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Thu, 25 Sep 2014 14:50:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1252) topic/jsiwek/missing-plugin
In-Reply-To: <JIRA.16000.1410880686000@Atlassian.JIRA>
References: <JIRA.16000.1410880686000@Atlassian.JIRA>
	<JIRA.16000.1410880686994@bro-tracker.atlassian.net>
Message-ID: <JIRA.16000.1410880686000.47.1411674607741@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robin Sommer updated BIT-1252:
------------------------------
    Resolution: Merged  (was: Fixed)
        Status: Closed  (was: Merge Request)

> topic/jsiwek/missing-plugin
> ---------------------------
>
>                 Key: BIT-1252
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1252
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>            Reporter: Jon Siwek
>            Assignee: Robin Sommer
>             Fix For: 2.4
>
>
> Trying to activate a non-existing dynamic plugin gave a hard to understand error message and core dumps:
> {noformat}
> $ bro Bro::NOPE
> error in /Users/jsiwek/Projects/bro/bro/scripts/base/init-bare.bro, line 1: plugin Bro::NOPE is not available
> internal error in /Users/jsiwek/Projects/bro/bro/build/scripts/base/bif/const.bif.bro, line 33: internal variable state_dir missing
> Abort trap: 6
> {noformat}
> It also seems that {{bro -N Bro::NOPE}}, which is used by the {{testing/scripts/has-writer}} helper script, is interpreted as a request to activate a dynamic plugin instead of just query if a plugin exists w/ that name.  Not sure the intention: I just changed the helper script to grep the output of {{bro -N}} instead.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 25 12:50:08 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Thu, 25 Sep 2014 14:50:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1260) topic/jsiwek/improve_comm_loop
In-Reply-To: <JIRA.16103.1411411475000@Atlassian.JIRA>
References: <JIRA.16103.1411411475000@Atlassian.JIRA>
	<JIRA.16103.1411411475304@bro-tracker.atlassian.net>
Message-ID: <JIRA.16103.1411411475000.51.1411674608269@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robin Sommer updated BIT-1260:
------------------------------
    Status: Open  (was: Merge Request)

> topic/jsiwek/improve_comm_loop
> ------------------------------
>
>                 Key: BIT-1260
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1260
>             Project: Bro Issue Tracker
>          Issue Type: Improvement
>          Components: Bro
>            Reporter: Jon Siwek
>             Fix For: 2.4
>
>
> This branch removes timeouts from the remote communication loop and so reduces cpu usage during the times it's supposed to be idle.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 25 12:50:08 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Thu, 25 Sep 2014 14:50:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1260) topic/jsiwek/improve_comm_loop
In-Reply-To: <JIRA.16103.1411411475000@Atlassian.JIRA>
References: <JIRA.16103.1411411475000@Atlassian.JIRA>
	<JIRA.16103.1411411475304@bro-tracker.atlassian.net>
Message-ID: <JIRA.16103.1411411475000.49.1411674608213@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18209#comment-18209 ] 

Robin Sommer commented on BIT-1260:
-----------------------------------

Nice idea with the Flares. Merged, but questions:

- the pipe operations in the Flare methods worry me: shouldn't unexpected errors lead to abort, rather than potential endless loops?

- I'm confused about the precise semantics ExtraReadFDs(): it says *Read*, but ChunkedIOFD also adds the write flare, but ChunkedIOSSL doesn't?

- API suggestion: can we give ChunkedIO just a single method FDs() that combines Fd() and ExtraReadFDs()?

Leaving open for now.

> topic/jsiwek/improve_comm_loop
> ------------------------------
>
>                 Key: BIT-1260
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1260
>             Project: Bro Issue Tracker
>          Issue Type: Improvement
>          Components: Bro
>            Reporter: Jon Siwek
>             Fix For: 2.4
>
>
> This branch removes timeouts from the remote communication loop and so reduces cpu usage during the times it's supposed to be idle.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 25 14:05:07 2014
From: jira at bro-tracker.atlassian.net (hui (JIRA))
Date: Thu, 25 Sep 2014 16:05:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1263) Implementing three event handlers for
 supported data structure in Modbus Analyzer
In-Reply-To: <JIRA.16106.1411679070000@Atlassian.JIRA>
References: <JIRA.16106.1411679070000@Atlassian.JIRA>
	<JIRA.16106.1411679070816@bro-tracker.atlassian.net>
Message-ID: <JIRA.16106.1411679070000.52.1411679107645@Atlassian.JIRA>

hui created BIT-1263:
------------------------

             Summary: Implementing three event handlers for supported data structure in Modbus Analyzer
                 Key: BIT-1263
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1263
             Project: Bro Issue Tracker
          Issue Type: Improvement
          Components: Bro
            Reporter: hui
            Priority: Low


Three support data structures are defined in Modbus analyzer:
FileRecordRequest,
FileRecordResponse,
ReferenceWithData

Three event handlers are declared for them. 

The changes are already made and pushed into the branch:
topic/hui/modbus-events2




--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Thu Sep 25 17:55:07 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Thu, 25 Sep 2014 19:55:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1249) Trouble with pcap interfaces with
 colons in their name
In-Reply-To: <JIRA.15907.1410368617000@Atlassian.JIRA>
References: <JIRA.15907.1410368617000@Atlassian.JIRA>
	<JIRA.15907.1410368617282@bro-tracker.atlassian.net>
Message-ID: <JIRA.15907.1410368617000.53.1411692907649@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robin Sommer updated BIT-1249:
------------------------------
    Resolution: Fixed
        Status: Closed  (was: Open)

Prefix separator is now "%", i.e., {{-i netmap%eth0}}.

> Trouble with pcap interfaces with colons in their name
> ------------------------------------------------------
>
>                 Key: BIT-1249
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1249
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>            Reporter: Robin Sommer
>            Assignee: Robin Sommer
>
> Pcap interfaces can have colons in their name, which confuses the new packet source code, as it uses "xxx:" to identify the type of source to use (e.g., "dag0:0" isn't handed over to pcap; instead Bro looks for a plugin "dag0" to handle the interface "0".)
> As a work-around, it should work to use "-i pcap:dag:0".
> Possible fixes:
>   - we use a different prefix separator (e.g., "-i netmap!em3")
>   - fallback to pcap if prefix not known.
>   - make the work around the solution (i.e., if in doubt, specify pcap explicitly)
> Reported by John Donaldson.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Fri Sep 26 06:34:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Fri, 26 Sep 2014 08:34:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1257) Same file id generated for potentially
 different files
In-Reply-To: <JIRA.16100.1411377279000@Atlassian.JIRA>
References: <JIRA.16100.1411377279000@Atlassian.JIRA>
	<JIRA.16100.1411377279508@bro-tracker.atlassian.net>
Message-ID: <JIRA.16100.1411377279000.55.1411738447714@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18211#comment-18211 ] 

Jimmy Jones commented on BIT-1257:
----------------------------------

{noformat}
[ 30%] core.tunnels.gtp.outer_ip_frag ... failed
{noformat}
http.log resp_fuids different, as expected
{noformat}
[ 66%] scripts.base.frameworks.file-analysis.bifs.set_timeout_interval ... failed
{noformat}
http/206_example_b.pcap, multiple requests were being incorrectly merged into one, but don't quite understand the output before (or after!)
{noformat}
[ 67%] scripts.base.frameworks.file-analysis.http.partial-content ... failed
{noformat}
http/206_example_a.pcap, multiple requests were being incorrectly merged into one, are now split into two files with the correct number of bytes. Now get FILE_TIMEOUT events, not sure why.

> Same file id generated for potentially different files
> ------------------------------------------------------
>
>                 Key: BIT-1257
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1257
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: fa.bro, sample-samefileid.pcap
>
>
> Attached sample contains two HTTP downloads of the same URL from the same client, but there are no guarantees that the files is actually the same (no Etags etc - in this case it actually is the same, but lets pretend they were different...). However the file analysis framework seems to give the same file ID in file_name and file_chunk for both downloads.
> Think this is something to do with Range requests as doesn't happen if do "normal" HTTP requests.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Fri Sep 26 07:04:07 2014
From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA))
Date: Fri, 26 Sep 2014 09:04:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1257) Same file id generated for potentially
 different files
In-Reply-To: <JIRA.16100.1411377279000@Atlassian.JIRA>
References: <JIRA.16100.1411377279000@Atlassian.JIRA>
	<JIRA.16100.1411377279508@bro-tracker.atlassian.net>
Message-ID: <JIRA.16100.1411377279000.57.1411740247681@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18212#comment-18212 ] 

Seth Hall commented on BIT-1257:
--------------------------------

The tests that are merging multiple files into one are actually working exactly like they're supposed to.  With the change you made, you will end up with two chunks of the file if you enable extraction but if you leave it as it is you will end up with one file that just happened to be transferred over two connections and reassembled back into the single original file.

This is definitely an area where there isn't a right answer so we just have to go based on experience of what's happening in real traffic and we definitely see this sort of stuff in real traffic.  Also, if you don't like Bro's behavior, you can run your own script (without modifying any of the shipped scripts) that gives you the behavior you're looking for.  Did you understand my suggestion about doing your own get_file_handle function and registering that at the begging of this ticket?

> Same file id generated for potentially different files
> ------------------------------------------------------
>
>                 Key: BIT-1257
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1257
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: fa.bro, sample-samefileid.pcap
>
>
> Attached sample contains two HTTP downloads of the same URL from the same client, but there are no guarantees that the files is actually the same (no Etags etc - in this case it actually is the same, but lets pretend they were different...). However the file analysis framework seems to give the same file ID in file_name and file_chunk for both downloads.
> Think this is something to do with Range requests as doesn't happen if do "normal" HTTP requests.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Fri Sep 26 08:27:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Fri, 26 Sep 2014 10:27:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1260) topic/jsiwek/improve_comm_loop
In-Reply-To: <JIRA.16103.1411411475000@Atlassian.JIRA>
References: <JIRA.16103.1411411475000@Atlassian.JIRA>
	<JIRA.16103.1411411475304@bro-tracker.atlassian.net>
Message-ID: <JIRA.16103.1411411475000.58.1411745227650@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18213#comment-18213 ] 

Jon Siwek commented on BIT-1260:
--------------------------------

{quote}
the pipe operations in the Flare methods worry me: shouldn't unexpected errors lead to abort, rather than potential endless loops?
{quote}

Sure adding extra safety precaution shouldn't hurt much, I'll do that -- in this case the unexpected errors would really, truly be unexpected, but I guess we'll see what we get :)

{quote}
I'm confused about the precise semantics ExtraReadFDs(): it says Read, but ChunkedIOFD also adds the write flare, but ChunkedIOSSL doesn't?
{quote}

The signaling mechanism of a flare is to make sure something can be *read* from its pipe for as long as one wants the signal to be active, so that's where ExtraReadFDs() came from -- e.g. "integrate these in to the read fd set of your select call to check if there's work (either input or output) to be done".

For ChunkedIOFD, it has two queues.  One that fills up with input that it "over-read", but failed to process in one go and another for output that's waiting to be written.  At the cost of using a couple extra file descriptors, it was easiest to just use two independent flares to track whether we still have read-work versus write-work to do.  Could probably be written to use a general "work flare", but maybe trickier to get that right.

For ChunkedIOSSL, it has the queue for output that's waiting to be written, but it never "over-reads" -- the read-readiness is always covered by the SSL wrapped file descriptor which extracts exactly one record at a time.  That's why it has just the extra write-flare while the original FD is sufficient to cover checking for input.

Let me know if that doesn't sound right.

{quote}
API suggestion: can we give ChunkedIO just a single method FDs() that combines Fd() and ExtraReadFDs()?
{quote}

Wasn't sure if it would ever be important for one to distinguish what the "real" FD (e.g. socket, etc.) is versus other internal ones for prioritization reasons.  Maybe overthinking, but also probably this API isn't going to be travelled often anyway so I can see either way being fine?

> topic/jsiwek/improve_comm_loop
> ------------------------------
>
>                 Key: BIT-1260
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1260
>             Project: Bro Issue Tracker
>          Issue Type: Improvement
>          Components: Bro
>            Reporter: Jon Siwek
>             Fix For: 2.4
>
>
> This branch removes timeouts from the remote communication loop and so reduces cpu usage during the times it's supposed to be idle.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Fri Sep 26 10:54:07 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Fri, 26 Sep 2014 12:54:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1260) topic/jsiwek/improve_comm_loop
In-Reply-To: <JIRA.16103.1411411475000@Atlassian.JIRA>
References: <JIRA.16103.1411411475000@Atlassian.JIRA>
	<JIRA.16103.1411411475304@bro-tracker.atlassian.net>
Message-ID: <JIRA.16103.1411411475000.61.1411754047962@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robin Sommer updated BIT-1260:
------------------------------
    Resolution: Merged
        Status: Closed  (was: Open)

> topic/jsiwek/improve_comm_loop
> ------------------------------
>
>                 Key: BIT-1260
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1260
>             Project: Bro Issue Tracker
>          Issue Type: Improvement
>          Components: Bro
>            Reporter: Jon Siwek
>             Fix For: 2.4
>
>
> This branch removes timeouts from the remote communication loop and so reduces cpu usage during the times it's supposed to be idle.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Fri Sep 26 10:54:07 2014
From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA))
Date: Fri, 26 Sep 2014 12:54:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1260) topic/jsiwek/improve_comm_loop
In-Reply-To: <JIRA.16103.1411411475000@Atlassian.JIRA>
References: <JIRA.16103.1411411475000@Atlassian.JIRA>
	<JIRA.16103.1411411475304@bro-tracker.atlassian.net>
Message-ID: <JIRA.16103.1411411475000.60.1411754047912@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18214#comment-18214 ] 

Robin Sommer commented on BIT-1260:
-----------------------------------

{quote}
truly be unexpected, but I guess we'll see what we get 
{quote}

Ack. :)

{quote}
Let me know if that doesn't sound right.
{quote}

Got it, thanks.

{quote}
anyway so I can see either way being fine?
{quote}

Yeah, agree, doesn't matter much. Hopefully all this code will go away in the not too distance future.

> topic/jsiwek/improve_comm_loop
> ------------------------------
>
>                 Key: BIT-1260
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1260
>             Project: Bro Issue Tracker
>          Issue Type: Improvement
>          Components: Bro
>            Reporter: Jon Siwek
>             Fix For: 2.4
>
>
> This branch removes timeouts from the remote communication loop and so reduces cpu usage during the times it's supposed to be idle.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From gc355804 at ohio.edu  Fri Sep 26 12:33:25 2014
From: gc355804 at ohio.edu (Gilbert Clark)
Date: Fri, 26 Sep 2014 15:33:25 -0400
Subject: [Bro-Dev] Bro + real-time question
Message-ID: <5425BF85.5090403@ohio.edu>

Hi list:

Does anyone know of work that involves placing hard limits on the amount 
of time bro is able to spend processing individual packets? 
Specifically, I'm looking for:

* work toward putting hard limits on the number of cycles Bro is allowed 
to execute per-packet before injecting a hard stop and forcing the 
engine to move to the next packet, or
* work toward emulating buffers / drops based on the number of cycles 
bro spends processing a particular packet in offline / pseudo-realtime mode

Thanks in advance for any references / thoughts!

--Gilbert


From asharma at lbl.gov  Fri Sep 26 12:58:37 2014
From: asharma at lbl.gov (Aashish Sharma)
Date: Fri, 26 Sep 2014 12:58:37 -0700
Subject: [Bro-Dev] Bro + real-time question
In-Reply-To: <5425BF85.5090403@ohio.edu>
References: <5425BF85.5090403@ohio.edu>
Message-ID: <20140926195835.GC391@yaksha.lbl.gov>

> * work toward putting hard limits on the number of cycles Bro is allowed 
> to execute per-packet before injecting a hard stop and forcing the 
> engine to move to the next packet, or

I believe it the value of : watchdog_interval 

Aashish 


On Fri, Sep 26, 2014 at 03:33:25PM -0400, Gilbert Clark wrote:
> Hi list:
> 
> Does anyone know of work that involves placing hard limits on the amount 
> of time bro is able to spend processing individual packets? 
> Specifically, I'm looking for:
> 
> * work toward putting hard limits on the number of cycles Bro is allowed 
> to execute per-packet before injecting a hard stop and forcing the 
> engine to move to the next packet, or
> * work toward emulating buffers / drops based on the number of cycles 
> bro spends processing a particular packet in offline / pseudo-realtime mode
> 
> Thanks in advance for any references / thoughts!
> 
> --Gilbert
> 
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

-- 
Aashish Sharma	(asharma at lbl.gov) 				 
Cyber Security, 
Lawrence Berkeley National Laboratory  
http://go.lbl.gov/pgp-aashish 
Office: (510)-495-2680  Cell: (510)-612-7971
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140926/90639e7d/attachment.bin 

From noreply at bro.org  Sat Sep 27 00:00:24 2014
From: noreply at bro.org (Merge Tracker)
Date: Sat, 27 Sep 2014 00:00:24 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409270700.s8R70OBH006892@bro-ids.icir.org>


Open Fastpath Commits
======================

Commit       Component    Author     Date        Summary
-----------  -----------  ---------  ----------  ------------------------------------------------------------
f933899 [1]  bro          Hui Lin    2014-09-26  adding a function in dnp3-analyzer.pac to translate the time
57d0346 [2]  bro          Jon Siwek  2014-09-26  Make unexpected pipe errors fatal as precaution.


[1]  f933899  https://github.com/bro/bro/commit/f933899b172647c58779ed914612ba182612b659
[2]  57d0346  https://github.com/bro/bro/commit/57d0346789e9a15f0ca8a6e6761cc4994ad73cb6


From vern at icir.org  Sat Sep 27 12:43:03 2014
From: vern at icir.org (Vern Paxson)
Date: Sat, 27 Sep 2014 12:43:03 -0700
Subject: [Bro-Dev] Bro + real-time question
In-Reply-To: <20140926195835.GC391@yaksha.lbl.gov> (Fri,
	26 Sep 2014 12:58:37 PDT).
Message-ID: <20140927194303.2EC9B2C4018@rock.ICSI.Berkeley.EDU>

> I believe it the value of : watchdog_interval

Unless things have changed, that instead is a global timer which kills Bro
(rather than skipping to the next packet) if that much time has elapsed
and it hasn't gotten past the current packet..  The notion is that Bro has
become wedged and needs a hard recovery.

		Vern

From noreply at bro.org  Sun Sep 28 00:00:22 2014
From: noreply at bro.org (Merge Tracker)
Date: Sun, 28 Sep 2014 00:00:22 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409280700.s8S70M4G027599@bro-ids.icir.org>


Open Fastpath Commits
======================

Commit       Component    Author     Date        Summary
-----------  -----------  ---------  ----------  ------------------------------------------------------------
f933899 [1]  bro          Hui Lin    2014-09-26  adding a function in dnp3-analyzer.pac to translate the time
57d0346 [2]  bro          Jon Siwek  2014-09-26  Make unexpected pipe errors fatal as precaution.


[1]  f933899  https://github.com/bro/bro/commit/f933899b172647c58779ed914612ba182612b659
[2]  57d0346  https://github.com/bro/bro/commit/57d0346789e9a15f0ca8a6e6761cc4994ad73cb6


From vern at icir.org  Sun Sep 28 10:25:28 2014
From: vern at icir.org (Vern Paxson)
Date: Sun, 28 Sep 2014 10:25:28 -0700
Subject: [Bro-Dev] Bro + real-time question
In-Reply-To: <5425BF85.5090403@ohio.edu> (Fri, 26 Sep 2014 15:33:25 EDT).
Message-ID: <20140928172528.4F3EA2C400B@rock.ICSI.Berkeley.EDU>

> Does anyone know of work that involves placing hard limits on the amount 
> of time bro is able to spend processing individual packets? 

Can you sketch your use case?  Different concerns (in particular, adversarial
threats versus performance problems) have different implications.

		Vern

From gc355804 at ohio.edu  Sun Sep 28 13:20:04 2014
From: gc355804 at ohio.edu (Gilbert Clark)
Date: Sun, 28 Sep 2014 16:20:04 -0400
Subject: [Bro-Dev] Bro + real-time question
In-Reply-To: <20140927194303.2EC9B2C4018@rock.ICSI.Berkeley.EDU>
References: <20140927194303.2EC9B2C4018@rock.ICSI.Berkeley.EDU>
Message-ID: <54286D74.8000607@ohio.edu>

Aashish: not quite what I'm looking for, but I do appreciate the link!  
I would be looking for something with higher resolution (usec / msec) 
that doesn't stop bro when the timer fires.

Vern: yeah, that's still the case as far as I know.

Thanks all!

-Gilbert

On 9/27/2014 3:43 PM, Vern Paxson wrote:
>> I believe it the value of : watchdog_interval
> Unless things have changed, that instead is a global timer which kills Bro
> (rather than skipping to the next packet) if that much time has elapsed
> and it hasn't gotten past the current packet..  The notion is that Bro has
> become wedged and needs a hard recovery.
>
> 		Vern


From gc355804 at ohio.edu  Sun Sep 28 14:49:16 2014
From: gc355804 at ohio.edu (Gilbert Clark)
Date: Sun, 28 Sep 2014 17:49:16 -0400
Subject: [Bro-Dev] Bro + real-time question
In-Reply-To: <20140928172528.4F3EA2C400B@rock.ICSI.Berkeley.EDU>
References: <20140928172528.4F3EA2C400B@rock.ICSI.Berkeley.EDU>
Message-ID: <5428825C.6080902@ohio.edu>

I was thinking one way to gracefully address performance issues might be 
to say that bro would be allowed to spend X cycles processing a specific 
packet, where X was a number determined by examining e.g. the current 
state of the network buffer + the historical packet rate / size.  
Enforcing a cut-off after X cycles could provide a way to dynamically 
scale the depth of the analysis to cope with additional load in lieu of 
completely dropping packets.

Could be that this is a terrible idea, but was just doing some homework 
/ reading and thought I'd ask to see if anyone could point me to work 
along these lines (or possibly explain why the ideas are not good ones :).

Regardless, thank you for taking the time to follow up!

Cheers,
Gilbert

On 9/28/2014 1:25 PM, Vern Paxson wrote:
> Can you sketch your use case?  Different concerns (in particular, adversarial
> threats versus performance problems) have different implications.


From vern at icir.org  Sun Sep 28 16:57:22 2014
From: vern at icir.org (Vern Paxson)
Date: Sun, 28 Sep 2014 16:57:22 -0700
Subject: [Bro-Dev] Bro + real-time question
In-Reply-To: <5428825C.6080902@ohio.edu> (Sun, 28 Sep 2014 17:49:16 EDT).
Message-ID: <20140928235722.BB4882C4025@rock.ICSI.Berkeley.EDU>

For performance concerns, it's not clear that individual packets are the
right granularity to examine.  For example, if you stop processing one
packet you might be giving up on any subsequent analysis for the remainder
of its flow, which can have a large amplifying effect (or not) depending
on the size of the flow.

For a different approach to the problem, see section 5.3 ("Dynamically
controlling packet load") in the Operational Experiences paper,
http://www.icir.org/vern/papers/high-volume-ccs04.pdf .

		Vern

From gc355804 at ohio.edu  Sun Sep 28 20:54:26 2014
From: gc355804 at ohio.edu (Clark, Gilbert)
Date: Mon, 29 Sep 2014 03:54:26 +0000
Subject: [Bro-Dev] Bro + real-time question
In-Reply-To: <20140928235722.BB4882C4025@rock.ICSI.Berkeley.EDU>
References: <5428825C.6080902@ohio.edu> (Sun, 28 Sep 2014 17:49:16
	EDT).,<20140928235722.BB4882C4025@rock.ICSI.Berkeley.EDU>
Message-ID: <1411962865889.59059@ohio.edu>

That makes sense.  I'll go ahead and read through that paper.

Thanks for the reference!

-Gilbert
________________________________________
From: Vern Paxson <vern at ICIR.org>
Sent: Sunday, September 28, 2014 7:57 PM
To: Clark, Gilbert
Cc: bro-dev at bro.org
Subject: Re: [Bro-Dev] Bro + real-time question

For performance concerns, it's not clear that individual packets are the
right granularity to examine.  For example, if you stop processing one
packet you might be giving up on any subsequent analysis for the remainder
of its flow, which can have a large amplifying effect (or not) depending
on the size of the flow.

For a different approach to the problem, see section 5.3 ("Dynamically
controlling packet load") in the Operational Experiences paper,
http://www.icir.org/vern/papers/high-volume-ccs04.pdf .

                Vern


From noreply at bro.org  Mon Sep 29 00:00:18 2014
From: noreply at bro.org (Merge Tracker)
Date: Mon, 29 Sep 2014 00:00:18 -0700
Subject: [Bro-Dev] [Auto] Merge Status
Message-ID: <201409290700.s8T70Iba013848@bro-ids.icir.org>


Open Fastpath Commits
======================

Commit       Component    Author         Date        Summary
-----------  -----------  -------------  ----------  ------------------------------------------------------------
470d868 [1]  bro          Johanna Amann  2014-09-28  new ssl extension type from iana and a few other ssl const c
f933899 [2]  bro          Hui Lin        2014-09-26  adding a function in dnp3-analyzer.pac to translate the time
57d0346 [3]  bro          Jon Siwek      2014-09-26  Make unexpected pipe errors fatal as precaution.


[1]  470d868  https://github.com/bro/bro/commit/470d86855822b362470f1b7eaaaf934b26a43d7e
[2]  f933899  https://github.com/bro/bro/commit/f933899b172647c58779ed914612ba182612b659
[3]  57d0346  https://github.com/bro/bro/commit/57d0346789e9a15f0ca8a6e6761cc4994ad73cb6


From jira at bro-tracker.atlassian.net  Mon Sep 29 02:19:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Mon, 29 Sep 2014 04:19:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1257) Same file id generated for potentially
 different files
In-Reply-To: <JIRA.16100.1411377279000@Atlassian.JIRA>
References: <JIRA.16100.1411377279000@Atlassian.JIRA>
	<JIRA.16100.1411377279508@bro-tracker.atlassian.net>
Message-ID: <JIRA.16100.1411377279000.64.1411982347736@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18215#comment-18215 ] 

Jimmy Jones commented on BIT-1257:
----------------------------------

Sorry I've not been as clear as I could here. I've changed my own bro instance, but I'm concerned that out of the box, Bro's behaviour while convenient for the majority of cases, isn't correct and will result in irrecoverably corrupted files in some instances (unless you?re lucky enough to keep full captures).

I've researched this further and I would argue there is a right answer and the spec is clear, see RFC2616, 10.2.7:

bq. A cache MUST NOT combine a 206 response with other previously cached content if the ETag or Last-Modified headers do not match exactly, see 13.5.4.

I'd say Bro is a cache in this instance, and for example clients like IE follow this [behavior|http://blogs.msdn.com/b/ieinternals/archive/2011/06/03/send-an-etag-to-enable-http-206-file-download-resume-without-restarting.aspx] and Adobe Reader uses the If-Range conditional to ensure the URL is the same document.

I agree my change is over-conservative, would you accept something that include ETag and Last-Modified in the hash? Or is the (small) chance of corruption not a concern (which is fine, as long as someone has actively decided not to follow the RFC)


> Same file id generated for potentially different files
> ------------------------------------------------------
>
>                 Key: BIT-1257
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1257
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: fa.bro, sample-samefileid.pcap
>
>
> Attached sample contains two HTTP downloads of the same URL from the same client, but there are no guarantees that the files is actually the same (no Etags etc - in this case it actually is the same, but lets pretend they were different...). However the file analysis framework seems to give the same file ID in file_name and file_chunk for both downloads.
> Think this is something to do with Range requests as doesn't happen if do "normal" HTTP requests.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)


From jira at bro-tracker.atlassian.net  Mon Sep 29 02:26:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Mon, 29 Sep 2014 04:26:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1264) HTTP response not detected on
	nonstandard port
In-Reply-To: <JIRA.16107.1411982737000@Atlassian.JIRA>
References: <JIRA.16107.1411982737000@Atlassian.JIRA>
	<JIRA.16107.1411982737931@bro-tracker.atlassian.net>
Message-ID: <JIRA.16107.1411982737000.65.1411982767626@Atlassian.JIRA>

Jimmy Jones created BIT-1264:
--------------------------------

             Summary: HTTP response not detected on nonstandard port
                 Key: BIT-1264
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1264
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: git/master
         Environment: CentOS 6
            Reporter: Jimmy Jones
         Attachments: relaxed.bro, relaxed-http.sig, sample-small2-rsp.pcap, sample-small-rsp.pcap

Using the attached bro script I've tweaked the HTTP signature to match on http responses without the corresponding HTTP request TCP session. I know in a proper setup you should never get single sided traffic, but certainly when using bro as a tool you have to deal with it sometimes.

Bro handles this fine when the HTTP is on port 80, but not when on port 4321 (see attached PCAPs). I'm curious as to why?



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 29 04:54:07 2014
From: jira at bro-tracker.atlassian.net (Jimmy Jones (JIRA))
Date: Mon, 29 Sep 2014 06:54:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1265) Single sided HTTP POST split
In-Reply-To: <JIRA.16108.1411991618000@Atlassian.JIRA>
References: <JIRA.16108.1411991618000@Atlassian.JIRA>
	<JIRA.16108.1411991618074@bro-tracker.atlassian.net>
Message-ID: <JIRA.16108.1411991618000.66.1411991647643@Atlassian.JIRA>

Jimmy Jones created BIT-1265:
--------------------------------

             Summary: Single sided HTTP POST split
                 Key: BIT-1265
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1265
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
    Affects Versions: git/master
         Environment: CentOS 6
            Reporter: Jimmy Jones
         Attachments: sample-upload2-all.pcap, sample-upload2-req.pcap

Attached two pcap samples, one is a single sided version of the other, an HTTP POST.

When I process the single sided version (sample-upload2-req) conn.log shows two sessions (the HTTP POST tcp connection that has been split) and http.log shows a partial upload. However processing the original sample (sample-upload2-all) everything is as expected - one connection in conn.log and a complete http.log

Are there any parameters I can tweak to make this work?



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 29 09:37:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Mon, 29 Sep 2014 11:37:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1264) HTTP response not detected on
	nonstandard port
In-Reply-To: <JIRA.16107.1411982737000@Atlassian.JIRA>
References: <JIRA.16107.1411982737000@Atlassian.JIRA>
	<JIRA.16107.1411982737931@bro-tracker.atlassian.net>
Message-ID: <JIRA.16107.1411982737000.68.1412008627717@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18216#comment-18216 ] 

Jon Siwek commented on BIT-1264:
--------------------------------

The difference here is in [likely_server_ports|https://www.bro.org/sphinx/scripts/base/init-bare.bro.html?highlight=likely_server_ports#id-likely_server_ports].

Because 80/tcp is in the likely_server_ports set, Bro correctly infers the packets belong to the responder, then your signature matches.

Because 4321/tcp isn't in the set, Bro thinks the packets are from the originator, then the signature doesn't match because it requires checking against the responder's payload.  And if you did force the signature to match by taking away the "is responder" condition, the HTTP analyzer would still ignore the content because it looks like data coming from the originator without having fully set up a TCP connection -- that's generally a situation where the current HTTP analyzer doesn't deal well.

> HTTP response not detected on nonstandard port
> ----------------------------------------------
>
>                 Key: BIT-1264
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1264
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: relaxed.bro, relaxed-http.sig, sample-small2-rsp.pcap, sample-small-rsp.pcap
>
>
> Using the attached bro script I've tweaked the HTTP signature to match on http responses without the corresponding HTTP request TCP session. I know in a proper setup you should never get single sided traffic, but certainly when using bro as a tool you have to deal with it sometimes.
> Bro handles this fine when the HTTP is on port 80, but not when on port 4321 (see attached PCAPs). I'm curious as to why?



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 29 10:09:07 2014
From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA))
Date: Mon, 29 Sep 2014 12:09:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1265) Single sided HTTP POST split
In-Reply-To: <JIRA.16108.1411991618000@Atlassian.JIRA>
References: <JIRA.16108.1411991618000@Atlassian.JIRA>
	<JIRA.16108.1411991618074@bro-tracker.atlassian.net>
Message-ID: <JIRA.16108.1411991618000.70.1412010547731@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18217#comment-18217 ] 

Jon Siwek commented on BIT-1265:
--------------------------------

[tcp_attempt_delay|https://www.bro.org/sphinx-git/scripts/base/init-bare.bro.html#id-tcp_attempt_delay] seems to be the relevant option.

Related: [connection_attempt|https://www.bro.org/sphinx-git/scripts/base/bif/plugins/Bro_TCP.events.bif.bro.html?highlight=connection_attempt#id-connection_attempt]

> Single sided HTTP POST split
> ----------------------------
>
>                 Key: BIT-1265
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1265
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: sample-upload2-all.pcap, sample-upload2-req.pcap
>
>
> Attached two pcap samples, one is a single sided version of the other, an HTTP POST.
> When I process the single sided version (sample-upload2-req) conn.log shows two sessions (the HTTP POST tcp connection that has been split) and http.log shows a partial upload. However processing the original sample (sample-upload2-all) everything is as expected - one connection in conn.log and a complete http.log
> Are there any parameters I can tweak to make this work?



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 29 23:02:08 2014
From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA))
Date: Tue, 30 Sep 2014 01:02:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1242) Logs: add page listing logs to
 /script-reference/index.html
In-Reply-To: <JIRA.15900.1410202655000@Atlassian.JIRA>
References: <JIRA.15900.1410202655000@Atlassian.JIRA>
	<JIRA.15900.1410202655434@bro-tracker.atlassian.net>
Message-ID: <JIRA.15900.1410202655000.72.1412056928137@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Thayer updated BIT-1242:
-------------------------------
    Description: 
Add a page to documentation listing the Bro log files.

There's already a page listing common log files: 
http://www.bro.org/sphinx/logs/index.html#common-log-files

This page could be moved to http://www.bro.org/sphinx/script-reference/index.html and be expanded to list all known log files.

If this goes well, Robin suggested adding a btest that heuristically checks if something changes with regards to which logs we have.

  was:
Add a page to documentation listing the Bro log files.

There's already a page listing common log files: 
http://www.bro.org/sphinx/logs/index.html#common-log-files

This page could be moved to http://www.bro.org/sphinx/script-reference/index.html and be expanded to list all known log files.

If this goes well, Robin suggested adding a btest that heuristically checks if something changes with regards to which logs we have.  I'll open a ticket for that when we're done.


> Logs: add page listing logs to /script-reference/index.html
> -----------------------------------------------------------
>
>                 Key: BIT-1242
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1242
>             Project: Bro Issue Tracker
>          Issue Type: Improvement
>          Components: Documentation, Website
>            Reporter: Jeannette Dopheide
>            Assignee: Jeannette Dopheide
>
> Add a page to documentation listing the Bro log files.
> There's already a page listing common log files: 
> http://www.bro.org/sphinx/logs/index.html#common-log-files
> This page could be moved to http://www.bro.org/sphinx/script-reference/index.html and be expanded to list all known log files.
> If this goes well, Robin suggested adding a btest that heuristically checks if something changes with regards to which logs we have.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Mon Sep 29 23:03:07 2014
From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA))
Date: Tue, 30 Sep 2014 01:03:07 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1242) Logs: add page listing logs to
 /script-reference/index.html
In-Reply-To: <JIRA.15900.1410202655000@Atlassian.JIRA>
References: <JIRA.15900.1410202655000@Atlassian.JIRA>
	<JIRA.15900.1410202655434@bro-tracker.atlassian.net>
Message-ID: <JIRA.15900.1410202655000.74.1412056987639@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Thayer updated BIT-1242:
-------------------------------
    Fix Version/s: 2.4

> Logs: add page listing logs to /script-reference/index.html
> -----------------------------------------------------------
>
>                 Key: BIT-1242
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1242
>             Project: Bro Issue Tracker
>          Issue Type: Improvement
>          Components: Documentation, Website
>            Reporter: Jeannette Dopheide
>            Assignee: Jeannette Dopheide
>             Fix For: 2.4
>
>
> Add a page to documentation listing the Bro log files.
> There's already a page listing common log files: 
> http://www.bro.org/sphinx/logs/index.html#common-log-files
> This page could be moved to http://www.bro.org/sphinx/script-reference/index.html and be expanded to list all known log files.
> If this goes well, Robin suggested adding a btest that heuristically checks if something changes with regards to which logs we have.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From seth at icir.org  Tue Sep 30 05:54:32 2014
From: seth at icir.org (Seth Hall)
Date: Tue, 30 Sep 2014 08:54:32 -0400
Subject: [Bro-Dev] Bro + real-time question
In-Reply-To: <20140928235722.BB4882C4025@rock.ICSI.Berkeley.EDU>
References: <20140928235722.BB4882C4025@rock.ICSI.Berkeley.EDU>
Message-ID: <B7C583CC-72E0-43E5-A37F-4892E1608A4C@icir.org>


On Sep 28, 2014, at 7:57 PM, Vern Paxson <vern at icir.org> wrote:

> For a different approach to the problem, see section 5.3 ("Dynamically
> controlling packet load") in the Operational Experiences paper,
> http://www.icir.org/vern/papers/high-volume-ccs04.pdf .

Ah!  I don't think I've actually read this paper before, but if my quick skim is correct, this paper is where the old load-level.bro script came from, right?  I've been wanting to bring that capability back in some form for a long time (and I've even written a couple of draft modules at various points in time).

 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From jira at bro-tracker.atlassian.net  Tue Sep 30 06:36:08 2014
From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA))
Date: Tue, 30 Sep 2014 08:36:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1257) Same file id generated for potentially
 different files
In-Reply-To: <JIRA.16100.1411377279000@Atlassian.JIRA>
References: <JIRA.16100.1411377279000@Atlassian.JIRA>
	<JIRA.16100.1411377279508@bro-tracker.atlassian.net>
Message-ID: <JIRA.16100.1411377279000.78.1412084168661@Atlassian.JIRA>


     [ https://bro-tracker.atlassian.net/browse/BIT-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Seth Hall reassigned BIT-1257:
------------------------------

    Assignee: Seth Hall

> Same file id generated for potentially different files
> ------------------------------------------------------
>
>                 Key: BIT-1257
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1257
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>            Assignee: Seth Hall
>         Attachments: fa.bro, sample-samefileid.pcap
>
>
> Attached sample contains two HTTP downloads of the same URL from the same client, but there are no guarantees that the files is actually the same (no Etags etc - in this case it actually is the same, but lets pretend they were different...). However the file analysis framework seems to give the same file ID in file_name and file_chunk for both downloads.
> Think this is something to do with Range requests as doesn't happen if do "normal" HTTP requests.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From jira at bro-tracker.atlassian.net  Tue Sep 30 06:36:08 2014
From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA))
Date: Tue, 30 Sep 2014 08:36:08 -0500 (CDT)
Subject: [Bro-Dev] [JIRA] (BIT-1257) Same file id generated for potentially
 different files
In-Reply-To: <JIRA.16100.1411377279000@Atlassian.JIRA>
References: <JIRA.16100.1411377279000@Atlassian.JIRA>
	<JIRA.16100.1411377279508@bro-tracker.atlassian.net>
Message-ID: <JIRA.16100.1411377279000.76.1412084168517@Atlassian.JIRA>


    [ https://bro-tracker.atlassian.net/browse/BIT-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18218#comment-18218 ] 

Seth Hall commented on BIT-1257:
--------------------------------

Ah!  I think it's perfectly reasonable to make our default behavior a bit closer to RFC2616.  I'll take a look into it soon.

At the very least, if someone does want the more liberal file combining they can add it back with a separate script (which I'll probably include with Bro somewhere).  I'll take this ticket to make sure I deal with this soon.

> Same file id generated for potentially different files
> ------------------------------------------------------
>
>                 Key: BIT-1257
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1257
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: fa.bro, sample-samefileid.pcap
>
>
> Attached sample contains two HTTP downloads of the same URL from the same client, but there are no guarantees that the files is actually the same (no Etags etc - in this case it actually is the same, but lets pretend they were different...). However the file analysis framework seems to give the same file ID in file_name and file_chunk for both downloads.
> Think this is something to do with Range requests as doesn't happen if do "normal" HTTP requests.



--
This message was sent by Atlassian JIRA
(v6.4-OD-05-009#64003)

From vern at icir.org  Tue Sep 30 09:10:51 2014
From: vern at icir.org (Vern Paxson)
Date: Tue, 30 Sep 2014 09:10:51 -0700
Subject: [Bro-Dev] Bro + real-time question
In-Reply-To: <B7C583CC-72E0-43E5-A37F-4892E1608A4C@icir.org> (Tue,
	30 Sep 2014 08:54:32 EDT).
Message-ID: <20140930161051.B05FF2C4053@rock.ICSI.Berkeley.EDU>

> ... but if my quick skim is correct, this paper is where the old
> load-level.bro script came from, right?

Yes, basically.  That paper documents a bunch of our experiences, including
load-level.  I think we wound up being torn between the need for it versus
the notion "if you need it, what you really need is a bigger cluster".

		Vern

From anthony.kasza at gmail.com  Tue Sep 30 10:08:10 2014
From: anthony.kasza at gmail.com (anthony kasza)
Date: Tue, 30 Sep 2014 10:08:10 -0700
Subject: [Bro-Dev] Bro + real-time question
In-Reply-To: <20140930161051.B05FF2C4053@rock.ICSI.Berkeley.EDU>
References: <B7C583CC-72E0-43E5-A37F-4892E1608A4C@icir.org>
	<20140930161051.B05FF2C4053@rock.ICSI.Berkeley.EDU>
Message-ID: <CAEZw2bzPop_ELASrixN5i3npCGd9=n3+AorAF=Cp=Q_D1ZCAbg@mail.gmail.com>

There are tons of incredibly useful features in Bro. Its always interesting
to see the research papers those features came from. I'm referring to table
expiring attributes in this case.

Its also interesting to read that connection timers are only disabled at
connection state removal. Are they periodically removed similar to hash
table resizing in figure 3? In any case, it's smart that their removal
comes second to packet processing.

-AK
On Sep 30, 2014 9:17 AM, "Vern Paxson" <vern at icir.org> wrote:

> > ... but if my quick skim is correct, this paper is where the old
> > load-level.bro script came from, right?
>
> Yes, basically.  That paper documents a bunch of our experiences, including
> load-level.  I think we wound up being torn between the need for it versus
> the notion "if you need it, what you really need is a bigger cluster".
>
>                 Vern
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140930/c91142f8/attachment.html 

From robin at icir.org  Tue Sep 30 11:10:10 2014
From: robin at icir.org (Robin Sommer)
Date: Tue, 30 Sep 2014 11:10:10 -0700
Subject: [Bro-Dev] Bro + real-time question
In-Reply-To: <CAEZw2bzPop_ELASrixN5i3npCGd9=n3+AorAF=Cp=Q_D1ZCAbg@mail.gmail.com>
References: <B7C583CC-72E0-43E5-A37F-4892E1608A4C@icir.org>
	<20140930161051.B05FF2C4053@rock.ICSI.Berkeley.EDU>
	<CAEZw2bzPop_ELASrixN5i3npCGd9=n3+AorAF=Cp=Q_D1ZCAbg@mail.gmail.com>
Message-ID: <20140930181010.GK76556@icir.org>



On Tue, Sep 30, 2014 at 10:08 -0700, you wrote:

> Its also interesting to read that connection timers are only disabled at
> connection state removal. Are they periodically removed similar to hash
> table resizing in figure 3?

If I recall that correctly, they used to be only disabled, but then
the change was to actually now remove them immediately to avoid
accumulating stale timers in memory. A single removal is pretty fast
due to the underlying data structure, and these operations amortize
over time, with no need for something like the hash table resizing.

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 *     robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 * www.icir.org/robin