From noreply at bro.org Sun Dec 1 00:00:09 2013 From: noreply at bro.org (Merge Tracker) Date: Sun, 1 Dec 2013 00:00:09 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201312010800.rB1809Uh001035@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ------------ ---------- ---------- ---------- ------------- ---------- ------------------------- BIT-1098 [1] Bro,Broccoli Jon Siwek - 2013-11-25 2.3 Normal topic/jsiwek/broxygen [2] [1] BIT-1098 https://bro-tracker.atlassian.net/browse/BIT-1098 [2] broxygen https://github.com/bro/bro,broccoli/tree/topic/jsiwek/broxygen From noreply at bro.org Mon Dec 2 00:00:16 2013 From: noreply at bro.org (Merge Tracker) Date: Mon, 2 Dec 2013 00:00:16 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201312020800.rB280Gij015663@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ------------ ---------- ---------- ---------- ------------- ---------- ------------------------- BIT-1098 [1] Bro,Broccoli Jon Siwek - 2013-11-25 2.3 Normal topic/jsiwek/broxygen [2] [1] BIT-1098 https://bro-tracker.atlassian.net/browse/BIT-1098 [2] broxygen https://github.com/bro/bro,broccoli/tree/topic/jsiwek/broxygen From jira at bro-tracker.atlassian.net Mon Dec 2 10:33:45 2013 From: jira at bro-tracker.atlassian.net (tyler.schoenke (JIRA)) Date: Mon, 2 Dec 2013 12:33:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1099) ./build/bro-git-20131129/lib/broctl/plugins/lb_myricom.py In-Reply-To: References: Message-ID: tyler.schoenke created BIT-1099: ----------------------------------- Summary: ./build/bro-git-20131129/lib/broctl/plugins/lb_myricom.py Key: BIT-1099 URL: https://bro-tracker.atlassian.net/browse/BIT-1099 Project: Bro Issue Tracker Issue Type: Problem Components: BroControl Affects Versions: git/master Environment: RHEL6 Myricom 10G Reporter: tyler.schoenke Hi All, First, unrelated, thanks for fixing whatever was wrong with PktSrc.cc:222. I was core dumping on that with the Myricom 10G until I upgraded to the latest git version of Bro. I had been running with my Myricom environment variables configured in .bash_profile, but decided to move them to etc/node.cfg because that feature is now available in Bro 2.2. When I tried to start the workers, they failed and I got the message to run diag. When I ran diag, I noticed it showed the following. 28366 snf.0.-1 P (environ) SNF_DATARING_SIZE = 2147483648 (0x80000000) (2048.0 MiB) 28366 snf.0.-1 P (default) SNF_DESCRING_SIZE = 536870912 (0x20000000) (512.0 MiB) 28366 snf.0.-1 P (userset) SNF_FLAGS = 257 (0x101) 28366 snf.0.-1 P (environ) SNF_DEBUG_MASK = 3 (0x3) 28366 snf.0.-1 P (default) SNF_DEBUG_FILENAME = stderr Everything looks great except for SNF_FLAGS. Myricom defaults it to 0x1, and that was working when I ran bro from the command line. I found it was getting set in ./build/bro-git-20131129/lib/broctl/plugins/lb_myricom.py According to /opt/snf/include/snf.h, the only valid values for SNF_FLAGS are 0x1, 0x2, and 0x300. No possible combination of those will add up to 0x101. I'm guessing that is a typo. If not, please explain why it is defaulted to that, otherwise you may want to default to 0x1. Thanks, Tyler -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Mon Dec 2 10:38:45 2013 From: jira at bro-tracker.atlassian.net (tyler.schoenke (JIRA)) Date: Mon, 2 Dec 2013 12:38:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (TM-16) Index not working when traffic encapsulated in 802.1q trunk In-Reply-To: References: Message-ID: tyler.schoenke created TM-16: -------------------------------- Summary: Index not working when traffic encapsulated in 802.1q trunk Key: TM-16 URL: https://bro-tracker.atlassian.net/browse/TM-16 Project: Time Machine Issue Type: Problem Affects Versions: git/master Environment: Ubuntu 10.04 , pf_ring Reporter: tyler.schoenke Hi All, When I query the time machine index, I am not receiving any results. I just restarted time machine, and checked one of the recent class files to see there is traffic for a particular IP address. tcpdump -e -v -n -r class_all_1385406639.023206 "vlan and host 128.138.44.198" It shows some traffic, example: 128.138.44.198.54014 > 74.125.225.209.443: Flags [.], cksum 0x8d2c (correct), seq 1283940799:1283940800, ack 615539104, win 16311, length 1 19:11:00.571731632 10:8c:cf:57:46:00 > 00:1d:09:6a:d9:a9, ethertype 802.1Q (0x8100), length 70: vlan 987, p 0, ethertype IPv4, (tos 0x0, ttl 56, id 17482, offset 0, flags [none], proto TCP (6), length 52) When I telnet localhost 42042 and run the following command, I don't receive any results. query to_file "128.138.44.198.pcap" index ip "128.138.44.198" In the above tcpdump, you can see my traffic is 802.1Q trunked. I have to use the "vlan" BPF to extract it with tcpdump, and am wondering if the trunking is causing problems with indexing? I tested the same version of time machine on non-trunked traffic, and the index works fine. Let me know if you need any other configuration info. Tyler -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Mon Dec 2 16:58:45 2013 From: jira at bro-tracker.atlassian.net (grigorescu (JIRA)) Date: Mon, 2 Dec 2013 18:58:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1099) ./build/bro-git-20131129/lib/broctl/plugins/lb_myricom.py In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14900#comment-14900 ] grigorescu commented on BIT-1099: --------------------------------- Hi Tyler, First off - is this actually causing a problem for you? It's set this way to work with both SNF1 and SNF2. You're looking at SNF2, which doesn't define 0x100. SNF1 does: # If incoming data is to be partitioned across rings via RSS, an # alternative receive mode is to set the SNF_F_RX_PRIVATE=0x100 flag # which reduces the amount of shared SNF references at the cost of # an additional copy. This approach may provide better capture behavior # when the RSS distribution is unbalanced or more generally, when the # consumption rate of each process varies enough to cause large amounts # of packet drops. Here we set the SNF_F_RX_PRIVATE=0x100 flag with # SNF_F_PSHARED=0x1 flag to allow process-sharing. # export SNF_FLAGS=0x101 Setting this to 0x101 shouldn't affect SNF2 drivers; it will be like 0x1 to them. > ./build/bro-git-20131129/lib/broctl/plugins/lb_myricom.py > --------------------------------------------------------- > > Key: BIT-1099 > URL: https://bro-tracker.atlassian.net/browse/BIT-1099 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: git/master > Environment: RHEL6 > Myricom 10G > Reporter: tyler.schoenke > Labels: Myricom, RHEL6 > > Hi All, > First, unrelated, thanks for fixing whatever was wrong with PktSrc.cc:222. I was core dumping on that with the Myricom 10G until I upgraded to the latest git version of Bro. > I had been running with my Myricom environment variables configured in .bash_profile, but decided to move them to etc/node.cfg because that feature is now available in Bro 2.2. > When I tried to start the workers, they failed and I got the message to run diag. When I ran diag, I noticed it showed the following. > 28366 snf.0.-1 P (environ) SNF_DATARING_SIZE = 2147483648 (0x80000000) (2048.0 MiB) > 28366 snf.0.-1 P (default) SNF_DESCRING_SIZE = 536870912 (0x20000000) (512.0 MiB) > 28366 snf.0.-1 P (userset) SNF_FLAGS = 257 (0x101) > 28366 snf.0.-1 P (environ) SNF_DEBUG_MASK = 3 (0x3) > 28366 snf.0.-1 P (default) SNF_DEBUG_FILENAME = stderr > Everything looks great except for SNF_FLAGS. Myricom defaults it to 0x1, and that was working when I ran bro from the command line. > I found it was getting set in ./build/bro-git-20131129/lib/broctl/plugins/lb_myricom.py > According to /opt/snf/include/snf.h, the only valid values for SNF_FLAGS are 0x1, 0x2, and 0x300. No possible combination of those will add up to 0x101. I'm guessing that is a typo. If not, please explain why it is defaulted to that, otherwise you may want to default to 0x1. > Thanks, > Tyler -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Mon Dec 2 17:00:45 2013 From: jira at bro-tracker.atlassian.net (grigorescu (JIRA)) Date: Mon, 2 Dec 2013 19:00:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1099) ./build/bro-git-20131129/lib/broctl/plugins/lb_myricom.py In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14900#comment-14900 ] grigorescu edited comment on BIT-1099 at 12/2/13 6:59 PM: ---------------------------------------------------------- Hi Tyler, First off - is this actually causing a problem for you? It's set this way to work with both SNF1 and SNF2. You're looking at SNF2, which doesn't define 0x100. SNF1 does: {code} # If incoming data is to be partitioned across rings via RSS, an # alternative receive mode is to set the SNF_F_RX_PRIVATE=0x100 flag # which reduces the amount of shared SNF references at the cost of # an additional copy. This approach may provide better capture behavior # when the RSS distribution is unbalanced or more generally, when the # consumption rate of each process varies enough to cause large amounts # of packet drops. Here we set the SNF_F_RX_PRIVATE=0x100 flag with # SNF_F_PSHARED=0x1 flag to allow process-sharing. # export SNF_FLAGS=0x101 {code} Setting this to 0x101 shouldn't affect SNF2 drivers; it will be like 0x1 to them. was (Author: grigorescu): Hi Tyler, First off - is this actually causing a problem for you? It's set this way to work with both SNF1 and SNF2. You're looking at SNF2, which doesn't define 0x100. SNF1 does: {quote} # If incoming data is to be partitioned across rings via RSS, an # alternative receive mode is to set the SNF_F_RX_PRIVATE=0x100 flag # which reduces the amount of shared SNF references at the cost of # an additional copy. This approach may provide better capture behavior # when the RSS distribution is unbalanced or more generally, when the # consumption rate of each process varies enough to cause large amounts # of packet drops. Here we set the SNF_F_RX_PRIVATE=0x100 flag with # SNF_F_PSHARED=0x1 flag to allow process-sharing. # export SNF_FLAGS=0x101 {quote} Setting this to 0x101 shouldn't affect SNF2 drivers; it will be like 0x1 to them. > ./build/bro-git-20131129/lib/broctl/plugins/lb_myricom.py > --------------------------------------------------------- > > Key: BIT-1099 > URL: https://bro-tracker.atlassian.net/browse/BIT-1099 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: git/master > Environment: RHEL6 > Myricom 10G > Reporter: tyler.schoenke > Labels: Myricom, RHEL6 > > Hi All, > First, unrelated, thanks for fixing whatever was wrong with PktSrc.cc:222. I was core dumping on that with the Myricom 10G until I upgraded to the latest git version of Bro. > I had been running with my Myricom environment variables configured in .bash_profile, but decided to move them to etc/node.cfg because that feature is now available in Bro 2.2. > When I tried to start the workers, they failed and I got the message to run diag. When I ran diag, I noticed it showed the following. > 28366 snf.0.-1 P (environ) SNF_DATARING_SIZE = 2147483648 (0x80000000) (2048.0 MiB) > 28366 snf.0.-1 P (default) SNF_DESCRING_SIZE = 536870912 (0x20000000) (512.0 MiB) > 28366 snf.0.-1 P (userset) SNF_FLAGS = 257 (0x101) > 28366 snf.0.-1 P (environ) SNF_DEBUG_MASK = 3 (0x3) > 28366 snf.0.-1 P (default) SNF_DEBUG_FILENAME = stderr > Everything looks great except for SNF_FLAGS. Myricom defaults it to 0x1, and that was working when I ran bro from the command line. > I found it was getting set in ./build/bro-git-20131129/lib/broctl/plugins/lb_myricom.py > According to /opt/snf/include/snf.h, the only valid values for SNF_FLAGS are 0x1, 0x2, and 0x300. No possible combination of those will add up to 0x101. I'm guessing that is a typo. If not, please explain why it is defaulted to that, otherwise you may want to default to 0x1. > Thanks, > Tyler -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Mon Dec 2 17:00:45 2013 From: jira at bro-tracker.atlassian.net (grigorescu (JIRA)) Date: Mon, 2 Dec 2013 19:00:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1099) ./build/bro-git-20131129/lib/broctl/plugins/lb_myricom.py In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14900#comment-14900 ] grigorescu edited comment on BIT-1099 at 12/2/13 6:59 PM: ---------------------------------------------------------- Hi Tyler, First off - is this actually causing a problem for you? It's set this way to work with both SNF1 and SNF2. You're looking at SNF2, which doesn't define 0x100. SNF1 does: {quote} # If incoming data is to be partitioned across rings via RSS, an # alternative receive mode is to set the SNF_F_RX_PRIVATE=0x100 flag # which reduces the amount of shared SNF references at the cost of # an additional copy. This approach may provide better capture behavior # when the RSS distribution is unbalanced or more generally, when the # consumption rate of each process varies enough to cause large amounts # of packet drops. Here we set the SNF_F_RX_PRIVATE=0x100 flag with # SNF_F_PSHARED=0x1 flag to allow process-sharing. # export SNF_FLAGS=0x101 {quote} Setting this to 0x101 shouldn't affect SNF2 drivers; it will be like 0x1 to them. was (Author: grigorescu): Hi Tyler, First off - is this actually causing a problem for you? It's set this way to work with both SNF1 and SNF2. You're looking at SNF2, which doesn't define 0x100. SNF1 does: # If incoming data is to be partitioned across rings via RSS, an # alternative receive mode is to set the SNF_F_RX_PRIVATE=0x100 flag # which reduces the amount of shared SNF references at the cost of # an additional copy. This approach may provide better capture behavior # when the RSS distribution is unbalanced or more generally, when the # consumption rate of each process varies enough to cause large amounts # of packet drops. Here we set the SNF_F_RX_PRIVATE=0x100 flag with # SNF_F_PSHARED=0x1 flag to allow process-sharing. # export SNF_FLAGS=0x101 Setting this to 0x101 shouldn't affect SNF2 drivers; it will be like 0x1 to them. > ./build/bro-git-20131129/lib/broctl/plugins/lb_myricom.py > --------------------------------------------------------- > > Key: BIT-1099 > URL: https://bro-tracker.atlassian.net/browse/BIT-1099 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: git/master > Environment: RHEL6 > Myricom 10G > Reporter: tyler.schoenke > Labels: Myricom, RHEL6 > > Hi All, > First, unrelated, thanks for fixing whatever was wrong with PktSrc.cc:222. I was core dumping on that with the Myricom 10G until I upgraded to the latest git version of Bro. > I had been running with my Myricom environment variables configured in .bash_profile, but decided to move them to etc/node.cfg because that feature is now available in Bro 2.2. > When I tried to start the workers, they failed and I got the message to run diag. When I ran diag, I noticed it showed the following. > 28366 snf.0.-1 P (environ) SNF_DATARING_SIZE = 2147483648 (0x80000000) (2048.0 MiB) > 28366 snf.0.-1 P (default) SNF_DESCRING_SIZE = 536870912 (0x20000000) (512.0 MiB) > 28366 snf.0.-1 P (userset) SNF_FLAGS = 257 (0x101) > 28366 snf.0.-1 P (environ) SNF_DEBUG_MASK = 3 (0x3) > 28366 snf.0.-1 P (default) SNF_DEBUG_FILENAME = stderr > Everything looks great except for SNF_FLAGS. Myricom defaults it to 0x1, and that was working when I ran bro from the command line. > I found it was getting set in ./build/bro-git-20131129/lib/broctl/plugins/lb_myricom.py > According to /opt/snf/include/snf.h, the only valid values for SNF_FLAGS are 0x1, 0x2, and 0x300. No possible combination of those will add up to 0x101. I'm guessing that is a typo. If not, please explain why it is defaulted to that, otherwise you may want to default to 0x1. > Thanks, > Tyler -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From noreply at bro.org Tue Dec 3 00:00:19 2013 From: noreply at bro.org (Merge Tracker) Date: Tue, 3 Dec 2013 00:00:19 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201312030800.rB380Joe024108@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ------------ ---------- ---------- ---------- ------------- ---------- ------------------------- BIT-1098 [1] Bro,Broccoli Jon Siwek - 2013-11-25 2.3 Normal topic/jsiwek/broxygen [2] [1] BIT-1098 https://bro-tracker.atlassian.net/browse/BIT-1098 [2] broxygen https://github.com/bro/bro,broccoli/tree/topic/jsiwek/broxygen From robin at icir.org Tue Dec 3 08:09:54 2013 From: robin at icir.org (Robin Sommer) Date: Tue, 3 Dec 2013 08:09:54 -0800 Subject: [Bro-Dev] Fwd: [REL - 10amd64-default][security/bro] Failed for bro-2.2 in build In-Reply-To: <529A961A.4090809@ee.lbl.gov> References: <201311300808.rAU88xFv086973@beefy2.isc.freebsd.org> <529A961A.4090809@ee.lbl.gov> Message-ID: <20131203160954.GG52843@icir.org> Which clang version is this? I've tried it with a recent version of the clang 3.4 release branch, and that works fine for me. But based on the error message, I'm attaching a patch; does that help by any chance? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin -------------- next part -------------- diff --git a/src/logging/writers/SQLite.cc b/src/logging/writers/SQLite.cc index 46d1f17..25f5cb0 100644 --- a/src/logging/writers/SQLite.cc +++ b/src/logging/writers/SQLite.cc @@ -126,7 +126,7 @@ bool SQLite::DoInit(const WriterInfo& info, int arg_num_fields, fullpath.append(".sqlite"); string tablename; - map::const_iterator it = info.config.find("tablename"); + WriterInfo::config_map::const_iterator it = info.config.find("tablename"); if ( it == info.config.end() ) { MsgThread::Info(Fmt("tablename configuration option not found. Defaulting to path %s", info.path)); From jira at bro-tracker.atlassian.net Tue Dec 3 09:35:45 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 3 Dec 2013 11:35:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1100) topic/jsiwek/broccoli-vectors In-Reply-To: References: Message-ID: Jon Siwek created BIT-1100: ------------------------------ Summary: topic/jsiwek/broccoli-vectors Key: BIT-1100 URL: https://bro-tracker.atlassian.net/browse/BIT-1100 Project: Bro Issue Tracker Issue Type: New Feature Components: Bro, Broccoli Affects Versions: git/master Reporter: Jon Siwek Fix For: 2.3 This branch is in the bro and broccoli repos and adds support for broccoli clients to receive events that have arguments w/ vector values. Sending events that have arguments w/ vector values is still unsupported. (Broccoli generally seems to be limited in the complexity of types it can create compared to Bro). -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Tue Dec 3 09:35:45 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 3 Dec 2013 11:35:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1100) topic/jsiwek/broccoli-vectors In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1100: --------------------------- Status: Merge Request (was: Open) > topic/jsiwek/broccoli-vectors > ----------------------------- > > Key: BIT-1100 > URL: https://bro-tracker.atlassian.net/browse/BIT-1100 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro, Broccoli > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > This branch is in the bro and broccoli repos and adds support for broccoli clients to receive events that have arguments w/ vector values. > Sending events that have arguments w/ vector values is still unsupported. (Broccoli generally seems to be limited in the complexity of types it can create compared to Bro). -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From robin at icir.org Tue Dec 3 09:41:21 2013 From: robin at icir.org (Robin Sommer) Date: Tue, 3 Dec 2013 09:41:21 -0800 Subject: [Bro-Dev] [Bro-Commits] [git/broccoli] topic/jsiwek/broccoli-vectors: Add support for consuming events w/ vector args. (de39868) In-Reply-To: <201312031734.rB3HYbDw030226@bro-ids.icir.org> References: <201312031734.rB3HYbDw030226@bro-ids.icir.org> Message-ID: <20131203174121.GC69466@icir.org> What's the motivation for adding vector support? Is that needed for something specific? On Tue, Dec 03, 2013 at 09:34 -0800, Jonathan Siwek wrote: > a user would need. And that doesn't seem worth getting in to for now. Agreed, I don't think it's worth spending too much time on beefing up Broccoli further given that we want to replace it anyways. Also, Broccoli can't recreate pointer structures, as Bro does; which is more important for compound types. That's one part of the reason why support isn't there. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From robin at icir.org Tue Dec 3 10:07:46 2013 From: robin at icir.org (Robin Sommer) Date: Tue, 3 Dec 2013 10:07:46 -0800 Subject: [Bro-Dev] Proposed IOSource reorg Message-ID: <20131203180746.GD69466@icir.org> I'm thinking to move the IOSource infrastructure into its own subdirectory/namespace and turn the IOSourceRegistry into iosource::Manager in alignment with the layout we've started to move to with the logging/input/etc. I'd then move the classes derived from IOSource into corresponding subdirectories, like this: src/iosource/ src/iosource/Manager.{h,cc} src/iosource/IOSource.{h,cc} src/iosource/sources/pkt-src/PktSrc.{h,cc} src/iosource/sources/pkt-src/bpf/* src/iosource/sources/flow-src/* src/iosource/sources/dns-mgr/* src/iosource/sources/remote-serializer/* The sources would turn into plugin components. New types of packet sources (like netmap) would then go into iosource/pkt-src/foo/. Does that make sense? One piece where I'm unsure: would it be better to keep the remote serializer out if this and instead do a separate serializer/ hierarchy where all the current serialization/communication code goes? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From jsiwek at illinois.edu Tue Dec 3 10:09:12 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Tue, 3 Dec 2013 18:09:12 +0000 Subject: [Bro-Dev] [Bro-Commits] [git/broccoli] topic/jsiwek/broccoli-vectors: Add support for consuming events w/ vector args. (de39868) In-Reply-To: <20131203174121.GC69466@icir.org> References: <201312031734.rB3HYbDw030226@bro-ids.icir.org> <20131203174121.GC69466@icir.org> Message-ID: <7E4C35BD-B933-4E51-A6BD-8407791AAAE2@illinois.edu> On Dec 3, 2013, at 11:41 AM, Robin Sommer wrote: > What's the motivation for adding vector support? Is that needed for > something specific? My response to the ?some events not received by broccoli? thread on the bro users list should clarify what precipitated it. Generally, it seems like some common bro types/events will now have vectors in them, and not fixing this could be a common pitfall for users. It didn?t look like it would take more than a day and a full broccoli replacement could be some time away, so working on it seemed worthwhile. - Jon From jsiwek at illinois.edu Tue Dec 3 10:40:05 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Tue, 3 Dec 2013 18:40:05 +0000 Subject: [Bro-Dev] Proposed IOSource reorg In-Reply-To: <20131203180746.GD69466@icir.org> References: <20131203180746.GD69466@icir.org> Message-ID: <9C96769F-DCE2-4F64-9E07-AA94CDEE5191@illinois.edu> On Dec 3, 2013, at 12:07 PM, Robin Sommer wrote: > One piece where I'm unsure: would it be better to keep the remote > serializer out if this and instead do a separate serializer/ hierarchy > where all the current serialization/communication code goes? Maybe best would be if the remote serializer code is refactored so the code that implements the IOSource interface lives in the iosource/ tree, while the code that implements Serializer interface lives in a separate serializer/ tree? Though taking the reorg and plugin adaptation one step at a time would make sense to just stick it in iosource for now and then later consider what, if any, pieces to pull out and put in serializer/. - Jon From scampbell at lbl.gov Tue Dec 3 11:08:03 2013 From: scampbell at lbl.gov (Scott Campbell) Date: Tue, 03 Dec 2013 13:08:03 -0600 Subject: [Bro-Dev] Proposed IOSource reorg In-Reply-To: <20131203180746.GD69466@icir.org> References: <20131203180746.GD69466@icir.org> Message-ID: <529E2C13.6070901@lbl.gov> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Would the input framework code be morphed into iosource/sources or continue living in it's own directory? Re breaking out communication and serialization into it's own place, it seems like it has a distinct function outside of i/o - it is used by i/o, but it can work outside of the core functionality. Making this a one step at a time reorg as Jon suggests might be a less complex and destructive way to go about that... cheers, scott On 12/3/13 12:07 PM, Robin Sommer wrote: > I'm thinking to move the IOSource infrastructure into its own > subdirectory/namespace and turn the IOSourceRegistry into > iosource::Manager in alignment with the layout we've started to > move to with the logging/input/etc. I'd then move the classes > derived from IOSource into corresponding subdirectories, like > this: > > src/iosource/ src/iosource/Manager.{h,cc} > src/iosource/IOSource.{h,cc} > src/iosource/sources/pkt-src/PktSrc.{h,cc} > src/iosource/sources/pkt-src/bpf/* src/iosource/sources/flow-src/* > src/iosource/sources/dns-mgr/* > src/iosource/sources/remote-serializer/* > > The sources would turn into plugin components. New types of packet > sources (like netmap) would then go into iosource/pkt-src/foo/. > > Does that make sense? > > One piece where I'm unsure: would it be better to keep the remote > serializer out if this and instead do a separate serializer/ > hierarchy where all the current serialization/communication code > goes? > > Robin > -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.19 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlKeLBMACgkQK2Plq8B7ZBy2qwCghjZ2uKm+qaIT8jXs9UaqBMAR 3akAnA670VZg52SJTWDhgQVz/FqPHEYE =jpTO -----END PGP SIGNATURE----- From jira at bro-tracker.atlassian.net Tue Dec 3 12:45:45 2013 From: jira at bro-tracker.atlassian.net (tyler.schoenke (JIRA)) Date: Tue, 3 Dec 2013 14:45:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1099) ./build/bro-git-20131129/lib/broctl/plugins/lb_myricom.py In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901#comment-14901 ] tyler.schoenke commented on BIT-1099: ------------------------------------- Hi Vlad, I thought it was causing the workers to not start, but I am unable to replicate the issue. I would say disregard. Thanks for the explanation of the SNF1 code. Tyler > ./build/bro-git-20131129/lib/broctl/plugins/lb_myricom.py > --------------------------------------------------------- > > Key: BIT-1099 > URL: https://bro-tracker.atlassian.net/browse/BIT-1099 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: git/master > Environment: RHEL6 > Myricom 10G > Reporter: tyler.schoenke > Labels: Myricom, RHEL6 > > Hi All, > First, unrelated, thanks for fixing whatever was wrong with PktSrc.cc:222. I was core dumping on that with the Myricom 10G until I upgraded to the latest git version of Bro. > I had been running with my Myricom environment variables configured in .bash_profile, but decided to move them to etc/node.cfg because that feature is now available in Bro 2.2. > When I tried to start the workers, they failed and I got the message to run diag. When I ran diag, I noticed it showed the following. > 28366 snf.0.-1 P (environ) SNF_DATARING_SIZE = 2147483648 (0x80000000) (2048.0 MiB) > 28366 snf.0.-1 P (default) SNF_DESCRING_SIZE = 536870912 (0x20000000) (512.0 MiB) > 28366 snf.0.-1 P (userset) SNF_FLAGS = 257 (0x101) > 28366 snf.0.-1 P (environ) SNF_DEBUG_MASK = 3 (0x3) > 28366 snf.0.-1 P (default) SNF_DEBUG_FILENAME = stderr > Everything looks great except for SNF_FLAGS. Myricom defaults it to 0x1, and that was working when I ran bro from the command line. > I found it was getting set in ./build/bro-git-20131129/lib/broctl/plugins/lb_myricom.py > According to /opt/snf/include/snf.h, the only valid values for SNF_FLAGS are 0x1, 0x2, and 0x300. No possible combination of those will add up to 0x101. I'm guessing that is a typo. If not, please explain why it is defaulted to that, otherwise you may want to default to 0x1. > Thanks, > Tyler -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Tue Dec 3 13:01:45 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 3 Dec 2013 15:01:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1097) Unexpected string indexing behavior In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1097: --------------------------- Status: Merge Request (was: Open) > Unexpected string indexing behavior > ----------------------------------- > > Key: BIT-1097 > URL: https://bro-tracker.atlassian.net/browse/BIT-1097 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Robin Sommer > > Playing with string indexing/slicing, I'm seeing some (I think) non-intuitive behavior: > {code} > global s = "012345"; > print "A"; > print s[1:-1]; > print s[1:-2]; > print s[1:-3]; > print s[1:-4]; > print s[1:-5]; > print s[1:-6]; > print s[1:-7]; > print s[1:-8]; > print s[1:-9]; > print ""; > print "B"; > print s[-1:-1]; > print s[-1:-2]; > print s[-1:-3]; > print s[-1:-4]; > {code} > This prints: > {code} > A > 12345 > 1234 > 123 > 12 > 1 > 12345 > 12345 > 12345 > B > 5 > 5 > 5 > {code} > I would instead have expected: > (1) A to print empty lines for all cases with the 2nd index <= -6? > (2) B to print empty lines for all cases with the 2nd index <= -2? > So, is this intentional? -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Tue Dec 3 13:01:45 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 3 Dec 2013 15:01:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1097) Unexpected string indexing behavior In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14902#comment-14902 ] Jon Siwek commented on BIT-1097: -------------------------------- Looked pretty wrong to me, too. See if topic/jsiwek/string-slicing-fix is better. > Unexpected string indexing behavior > ----------------------------------- > > Key: BIT-1097 > URL: https://bro-tracker.atlassian.net/browse/BIT-1097 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Robin Sommer > > Playing with string indexing/slicing, I'm seeing some (I think) non-intuitive behavior: > {code} > global s = "012345"; > print "A"; > print s[1:-1]; > print s[1:-2]; > print s[1:-3]; > print s[1:-4]; > print s[1:-5]; > print s[1:-6]; > print s[1:-7]; > print s[1:-8]; > print s[1:-9]; > print ""; > print "B"; > print s[-1:-1]; > print s[-1:-2]; > print s[-1:-3]; > print s[-1:-4]; > {code} > This prints: > {code} > A > 12345 > 1234 > 123 > 12 > 1 > 12345 > 12345 > 12345 > B > 5 > 5 > 5 > {code} > I would instead have expected: > (1) A to print empty lines for all cases with the 2nd index <= -6? > (2) B to print empty lines for all cases with the 2nd index <= -2? > So, is this intentional? -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From noreply at bro.org Wed Dec 4 00:00:14 2013 From: noreply at bro.org (Merge Tracker) Date: Wed, 4 Dec 2013 00:00:14 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201312040800.rB480EpF010021@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ------------ ------------ ---------- ---------- ------------- ---------- ----------------------------------- BIT-1100 [1] Bro,Broccoli Jon Siwek - 2013-12-03 2.3 Normal topic/jsiwek/broccoli-vectors [2] BIT-1098 [3] Bro,Broccoli Jon Siwek - 2013-11-25 2.3 Normal topic/jsiwek/broxygen [4] BIT-1097 [5] Bro Robin Sommer - 2013-12-03 - Normal Unexpected string indexing behavior Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- --------- ---------- -------------------------------------------------- 2ea6011 [6] bro Jon Siwek 2013-12-03 Improve a unit test involving 'when' conditionals. [1] BIT-1100 https://bro-tracker.atlassian.net/browse/BIT-1100 [2] broccoli-vectors https://github.com/bro/bro,broccoli/tree/topic/jsiwek/broccoli-vectors [3] BIT-1098 https://bro-tracker.atlassian.net/browse/BIT-1098 [4] broxygen https://github.com/bro/bro,broccoli/tree/topic/jsiwek/broxygen [5] BIT-1097 https://bro-tracker.atlassian.net/browse/BIT-1097 [6] 2ea6011 https://github.com/bro/bro/commit/2ea6011186e91dff37c639667b1604d837363908 From robin at icir.org Wed Dec 4 07:31:22 2013 From: robin at icir.org (Robin Sommer) Date: Wed, 4 Dec 2013 07:31:22 -0800 Subject: [Bro-Dev] [Bro-Commits] [git/broccoli] topic/jsiwek/broccoli-vectors: Add support for consuming events w/ vector args. (de39868) In-Reply-To: <7E4C35BD-B933-4E51-A6BD-8407791AAAE2@illinois.edu> References: <201312031734.rB3HYbDw030226@bro-ids.icir.org> <20131203174121.GC69466@icir.org> <7E4C35BD-B933-4E51-A6BD-8407791AAAE2@illinois.edu> Message-ID: <20131204153122.GL69466@icir.org> On Tue, Dec 03, 2013 at 18:09 +0000, you wrote: > and not fixing this could be a common pitfall for users. Ack, makes sense. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From robin at icir.org Wed Dec 4 07:33:16 2013 From: robin at icir.org (Robin Sommer) Date: Wed, 4 Dec 2013 07:33:16 -0800 Subject: [Bro-Dev] Proposed IOSource reorg In-Reply-To: <9C96769F-DCE2-4F64-9E07-AA94CDEE5191@illinois.edu> References: <20131203180746.GD69466@icir.org> <9C96769F-DCE2-4F64-9E07-AA94CDEE5191@illinois.edu> Message-ID: <20131204153316.GM69466@icir.org> On Tue, Dec 03, 2013 at 18:40 +0000, you wrote: > Maybe best would be if the remote serializer code is refactored so the > code that implements the IOSource interface lives in the iosource/ > tree, while the code that implements Serializer interface lives in a > separate serializer/ tree? Could be an option, though I'm not immediately sure how well it would split. But one step at a time sounds good in any case, so I'll go ahead with that and we can later see. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From robin at icir.org Wed Dec 4 07:36:03 2013 From: robin at icir.org (Robin Sommer) Date: Wed, 4 Dec 2013 07:36:03 -0800 Subject: [Bro-Dev] Proposed IOSource reorg In-Reply-To: <529E2C13.6070901@lbl.gov> References: <20131203180746.GD69466@icir.org> <529E2C13.6070901@lbl.gov> Message-ID: <20131204153603.GN69466@icir.org> On Tue, Dec 03, 2013 at 13:08 -0600, you wrote: > Would the input framework code be morphed into iosource/sources or > continue living in it's own directory? No, the input framework is separate. The threading manager could be affected (it's an IOSource as well, and the input framework uses it) but that's probably best left where it is for now as well (it's actually a similar question as with RemoteSerializer; didn't realize that yesterday). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From seth at icir.org Wed Dec 4 08:12:58 2013 From: seth at icir.org (Seth Hall) Date: Wed, 4 Dec 2013 11:12:58 -0500 Subject: [Bro-Dev] Proposed IOSource reorg In-Reply-To: <20131203180746.GD69466@icir.org> References: <20131203180746.GD69466@icir.org> Message-ID: <663C4ECE-3500-45DC-94B7-8962ABA8391E@icir.org> On Dec 3, 2013, at 1:07 PM, Robin Sommer wrote: > src/iosource/sources/flow-src/* To document our conversation from yesterday, flow-src should probably be thrown out and the netflow analyzer turned into a file analyzer. Extending the input framework to be able to open raw sockets would then enable us to create an input stream holding open a datagram socket and attach the netflow file analyzer to it. This would simplify the whole thing and make it possible to reuse the netflow analyzer code because we could yank netflow directly off the wire with it too (pending some analyzer infrastructure re-architecting). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20131204/c3ba09b0/attachment.bin From jira at bro-tracker.atlassian.net Wed Dec 4 11:26:45 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Wed, 4 Dec 2013 13:26:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1097) Unexpected string indexing behavior In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14903#comment-14903 ] Robin Sommer commented on BIT-1097: ----------------------------------- yeah, looks better, but I think it's now missing another piece: with the changed semantics for the end position, how does one get "everything until the end of the string"? Python has {{a[5:]}} for that. > Unexpected string indexing behavior > ----------------------------------- > > Key: BIT-1097 > URL: https://bro-tracker.atlassian.net/browse/BIT-1097 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Robin Sommer > > Playing with string indexing/slicing, I'm seeing some (I think) non-intuitive behavior: > {code} > global s = "012345"; > print "A"; > print s[1:-1]; > print s[1:-2]; > print s[1:-3]; > print s[1:-4]; > print s[1:-5]; > print s[1:-6]; > print s[1:-7]; > print s[1:-8]; > print s[1:-9]; > print ""; > print "B"; > print s[-1:-1]; > print s[-1:-2]; > print s[-1:-3]; > print s[-1:-4]; > {code} > This prints: > {code} > A > 12345 > 1234 > 123 > 12 > 1 > 12345 > 12345 > 12345 > B > 5 > 5 > 5 > {code} > I would instead have expected: > (1) A to print empty lines for all cases with the 2nd index <= -6? > (2) B to print empty lines for all cases with the 2nd index <= -2? > So, is this intentional? -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Wed Dec 4 11:40:45 2013 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Wed, 4 Dec 2013 13:40:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1101) Merge topic/bernhard/ssl_ciphers_vector In-Reply-To: References: Message-ID: Bernhard Amann created BIT-1101: ----------------------------------- Summary: Merge topic/bernhard/ssl_ciphers_vector Key: BIT-1101 URL: https://bro-tracker.atlassian.net/browse/BIT-1101 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.3 topic/bernhard/ssl_ciphers_vector changes ciphers in the ssl_client_hello from a set into a vector. This preserves the ordering of the cipher suites the client sent, allowing e.g. better client fingerprinting. -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Wed Dec 4 11:40:45 2013 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Wed, 4 Dec 2013 13:40:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1101) Merge topic/bernhard/ssl_ciphers_vector In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1101: -------------------------------- Status: Merge Request (was: Open) > Merge topic/bernhard/ssl_ciphers_vector > --------------------------------------- > > Key: BIT-1101 > URL: https://bro-tracker.atlassian.net/browse/BIT-1101 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Fix For: 2.3 > > > topic/bernhard/ssl_ciphers_vector changes ciphers in the ssl_client_hello from a set into a vector. This preserves the ordering of the cipher suites the client sent, allowing e.g. better client fingerprinting. -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Wed Dec 4 12:29:45 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 4 Dec 2013 14:29:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1097) Unexpected string indexing behavior In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14904#comment-14904 ] Jon Siwek commented on BIT-1097: -------------------------------- {quote} how does one get "everything until the end of the string"? Python has a[5:] for that. {quote} {{a[5:|5|]}} ? I.e. {{a[5:]}} is like {{a[5:len(a)]}} in Python. Probably wouldn't be hard to add the shorthand, I can take a look. > Unexpected string indexing behavior > ----------------------------------- > > Key: BIT-1097 > URL: https://bro-tracker.atlassian.net/browse/BIT-1097 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Robin Sommer > > Playing with string indexing/slicing, I'm seeing some (I think) non-intuitive behavior: > {code} > global s = "012345"; > print "A"; > print s[1:-1]; > print s[1:-2]; > print s[1:-3]; > print s[1:-4]; > print s[1:-5]; > print s[1:-6]; > print s[1:-7]; > print s[1:-8]; > print s[1:-9]; > print ""; > print "B"; > print s[-1:-1]; > print s[-1:-2]; > print s[-1:-3]; > print s[-1:-4]; > {code} > This prints: > {code} > A > 12345 > 1234 > 123 > 12 > 1 > 12345 > 12345 > 12345 > B > 5 > 5 > 5 > {code} > I would instead have expected: > (1) A to print empty lines for all cases with the 2nd index <= -6? > (2) B to print empty lines for all cases with the 2nd index <= -2? > So, is this intentional? -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Wed Dec 4 12:31:45 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 4 Dec 2013 14:31:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1097) Unexpected string indexing behavior In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14904#comment-14904 ] Jon Siwek edited comment on BIT-1097 at 12/4/13 2:30 PM: --------------------------------------------------------- {quote} how does one get "everything until the end of the string"? Python has a[5:] for that. {quote} {{a[5:|a|]}} ? I.e. {{a[5:]}} is like {{a[5:len(a)]}} in Python. Probably wouldn't be hard to add the shorthand, I can take a look. was (Author: jsiwek): {quote} how does one get "everything until the end of the string"? Python has a[5:] for that. {quote} {{a[5:|5|]}} ? I.e. {{a[5:]}} is like {{a[5:len(a)]}} in Python. Probably wouldn't be hard to add the shorthand, I can take a look. > Unexpected string indexing behavior > ----------------------------------- > > Key: BIT-1097 > URL: https://bro-tracker.atlassian.net/browse/BIT-1097 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Robin Sommer > > Playing with string indexing/slicing, I'm seeing some (I think) non-intuitive behavior: > {code} > global s = "012345"; > print "A"; > print s[1:-1]; > print s[1:-2]; > print s[1:-3]; > print s[1:-4]; > print s[1:-5]; > print s[1:-6]; > print s[1:-7]; > print s[1:-8]; > print s[1:-9]; > print ""; > print "B"; > print s[-1:-1]; > print s[-1:-2]; > print s[-1:-3]; > print s[-1:-4]; > {code} > This prints: > {code} > A > 12345 > 1234 > 123 > 12 > 1 > 12345 > 12345 > 12345 > B > 5 > 5 > 5 > {code} > I would instead have expected: > (1) A to print empty lines for all cases with the 2nd index <= -6? > (2) B to print empty lines for all cases with the 2nd index <= -2? > So, is this intentional? -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Wed Dec 4 13:15:45 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 4 Dec 2013 15:15:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1097) Unexpected string indexing behavior In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905#comment-14905 ] Jon Siwek commented on BIT-1097: -------------------------------- That's in the branch now. > Unexpected string indexing behavior > ----------------------------------- > > Key: BIT-1097 > URL: https://bro-tracker.atlassian.net/browse/BIT-1097 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Robin Sommer > > Playing with string indexing/slicing, I'm seeing some (I think) non-intuitive behavior: > {code} > global s = "012345"; > print "A"; > print s[1:-1]; > print s[1:-2]; > print s[1:-3]; > print s[1:-4]; > print s[1:-5]; > print s[1:-6]; > print s[1:-7]; > print s[1:-8]; > print s[1:-9]; > print ""; > print "B"; > print s[-1:-1]; > print s[-1:-2]; > print s[-1:-3]; > print s[-1:-4]; > {code} > This prints: > {code} > A > 12345 > 1234 > 123 > 12 > 1 > 12345 > 12345 > 12345 > B > 5 > 5 > 5 > {code} > I would instead have expected: > (1) A to print empty lines for all cases with the 2nd index <= -6? > (2) B to print empty lines for all cases with the 2nd index <= -2? > So, is this intentional? -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Wed Dec 4 13:56:45 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Wed, 4 Dec 2013 15:56:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1098) topic/jsiwek/broxygen In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906#comment-14906 ] Robin Sommer commented on BIT-1098: ----------------------------------- Excellent, I like the new way of feeding the Broxygen manager. I've merged it. Some additional suggestions: - what would you think about moving the global type alias table into BroType as a private static member, along with a few static accessor methods to manipulate it? - if there's a bro binary in PATH, it seems that "make doc" picks up that one instead of the one in {{src/build/bro}} - "make doc" shows {{received termination signal}} right before {{Running Sphinx}}. Is that expected? - reST Output: -- I'd remove the "Param" in the function descriptions (or is that a Sphinx thing?) -- If a package has scripts both w/ and wo/ a further description, they appear separately in the package overview; see, e.g., {{scripts/base/frameworks/packet-filter/index.html}} -- should we just omit {{__load__.bro}} in the package overviews? -- I like how the records now show fields that are redefined somewhere else. Two further thoughts: --- how about changing {{from redef in XXX}} to {{added if XXX is loaded}}. For a user, it might be more intuitive to talk about loading another script than just redef. --- for the script that does the redef, can we show the new fields there as well? For example, in {{scripts/policy/protocols/http/software-browser-plugins.bro.html}} it would show not only that {{HTTP::Info}} got redef, but also repeat the fields it added right there. Same for the enums. > topic/jsiwek/broxygen > --------------------- > > Key: BIT-1098 > URL: https://bro-tracker.atlassian.net/browse/BIT-1098 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, Broccoli > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > This branch is in the bro and broccoli repos and improves the automated script-reference documentation generation process, broxygen. > This should address issues in BIT-701 and BIT-751, let me know if there's anything not covered that's still relevant. > Highlights: > - Remove {{--doc-scripts}} and {{-Z}} options to toggle documentation mode -- the parser is now always instrumented to gather documentation from comments of the form "##", "##!", or "##<". > - Raw comments are available at runtime through several BIF functions: {{get_*_comments}}; > - Add {{--broxygen}} and {{-X}} options to toggle generating reST-format documentation output, driven by a config file argument. > - Add a "broxygen" Sphinx extension domain, allowing certain pieces of documentation to be generated on-the-fly via invoking a Bro process. Re-organized/cleaned up the Sphinx source tree in {{doc/}} to use this in some places. -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Wed Dec 4 17:39:45 2013 From: jira at bro-tracker.atlassian.net (April Lorenzen (JIRA)) Date: Wed, 4 Dec 2013 19:39:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1102) Logs use tab delimiter yet do not remove tabs from logged data In-Reply-To: References: Message-ID: April Lorenzen created BIT-1102: ----------------------------------- Summary: Logs use tab delimiter yet do not remove tabs from logged data Key: BIT-1102 URL: https://bro-tracker.atlassian.net/browse/BIT-1102 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.2 Environment: ubuntu 12.04LTS server Reporter: April Lorenzen Attachments: 4872c1c6d6983c4b6cf4e0f83baf5ec5.pcap The attached pcap processed with bro -r pcapname will demonstrate the issue. View the http.log after processing and note that the field delimiter appears within a field. I tried searching the official github mirror but don't know bro well enough to find the spot to look. I'd be happy to work on fixing this if you point me to the problem area (that's the hard part, right?) -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Wed Dec 4 19:47:45 2013 From: jira at bro-tracker.atlassian.net (Andrew Hoying (JIRA)) Date: Wed, 4 Dec 2013 21:47:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1103) Memory leak in Bro Intel framework In-Reply-To: References: Message-ID: Andrew Hoying created BIT-1103: ---------------------------------- Summary: Memory leak in Bro Intel framework Key: BIT-1103 URL: https://bro-tracker.atlassian.net/browse/BIT-1103 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.2 Environment: Red Hat Enterprise Linux Server release 6.5 Reporter: Andrew Hoying Priority: High The policy/frameworks/intel/seen bro scripts have a memory leak. On my moderately busy Bro installation I am leaking about a gig of memory a day per worker process with the Intel framework enabled. I can replicate by adding the following to the local.bro default script and then running through a small PCAP with primarily dns, dhcp and syslog traffic. {{ @load policy/frameworks/intel/seen redef Intel::read_files += { "/usr/local/bro/spool/domain_suspicious.txt", }; }} The intel file is in the following format, here's a few sample lines. It is generated automatically by CIF: {{ #fields indicator indicator_type meta.source meta.desc meta.url meta.cif_impact meta.cif_severity meta.cif_confidence mete-tools.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=mete-tools.biz (public) - - 95 rttvxygkmwlqmq.net Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=rttvxygkmwlqmq.net (public) - - 95 podserveruho.com Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=podserveruho.com (public) - - 95 wwfcogdgntlxw.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=wwfcogdgntlxw.biz (public) - - 95 }} I compiled bro with gperftool debug support and followed the instructions here: http://www.bro.org/development/howtos/leaks.html. (Note, the instructions are wrong on the flags for ./configure, you need to add --enable-perftools-debug to get the -m option for bro) Here's the output from pprof top after running a PCAP trace with 10,000 packets. Running traces with more packets show a greater number of lost objects in the same code locations. {{ # pprof bin/bro "/tmp/bro.24541.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 Using local file bin/bro. Using local file /tmp/bro.24541.net_run-end.heap. Welcome to pprof! For help, type 'help'. (pprof) top Total: 4295 objects 2150 50.1% 50.1% 2150 50.1% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:186 2141 49.8% 99.9% 2141 49.8% copy_string /usr/src/bro-2.2/src/util.cc:155 2 0.0% 100.0% 2 0.0% re_alloc /usr/src/bro-2.2/build/src/re-scan.cc:2287 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:110 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:133 0 0.0% 100.0% 2141 49.8% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:195 0 0.0% 100.0% 4 0.1% Connection::NextPacket /usr/src/bro-2.2/src/Conn.cc:259 0 0.0% 100.0% 4 0.1% NetSessions::DispatchPacket /usr/src/bro-2.2/src/Sessions.cc:189 0 0.0% 100.0% 4 0.1% NetSessions::DoNextPacket /usr/src/bro-2.2/src/Sessions.cc:709 0 0.0% 100.0% 4 0.1% NetSessions::NextPacket /usr/src/bro-2.2/src/Sessions.cc:247 }} -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Wed Dec 4 20:16:45 2013 From: jira at bro-tracker.atlassian.net (Andrew Hoying (JIRA)) Date: Wed, 4 Dec 2013 22:16:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1103) Memory leak in Bro Intel framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907#comment-14907 ] Andrew Hoying commented on BIT-1103: ------------------------------------ I ran some more checks and the memory leak is only based on the size and frequency of the files loaded through Intel::read_files, the size and content of the trace files tested doesn't seem to make much difference in the final number of lost objects. > Memory leak in Bro Intel framework > ---------------------------------- > > Key: BIT-1103 > URL: https://bro-tracker.atlassian.net/browse/BIT-1103 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Environment: Red Hat Enterprise Linux Server release 6.5 > Reporter: Andrew Hoying > Priority: High > Labels: intel, leak > > The policy/frameworks/intel/seen bro scripts have a memory leak. On my moderately busy Bro installation I am leaking about a gig of memory a day per worker process with the Intel framework enabled. I can replicate by adding the following to the local.bro default script and then running through a small PCAP with primarily dns, dhcp and syslog traffic. > {{ > @load policy/frameworks/intel/seen > redef Intel::read_files += { > "/usr/local/bro/spool/domain_suspicious.txt", > }; > }} > The intel file is in the following format, here's a few sample lines. It is generated automatically by CIF: > {{ > #fields indicator indicator_type meta.source meta.desc meta.url meta.cif_impact meta.cif_severity meta.cif_confidence > mete-tools.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=mete-tools.biz (public) - - 95 > rttvxygkmwlqmq.net Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=rttvxygkmwlqmq.net (public) - - 95 > podserveruho.com Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=podserveruho.com (public) - - 95 > wwfcogdgntlxw.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=wwfcogdgntlxw.biz (public) - - 95 > }} > I compiled bro with gperftool debug support and followed the instructions here: http://www.bro.org/development/howtos/leaks.html. (Note, the instructions are wrong on the flags for ./configure, you need to add --enable-perftools-debug to get the -m option for bro) > Here's the output from pprof top after running a PCAP trace with 10,000 packets. Running traces with more packets show a greater number of lost objects in the same code locations. > {{ > # pprof bin/bro "/tmp/bro.24541.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 > Using local file bin/bro. > Using local file /tmp/bro.24541.net_run-end.heap. > Welcome to pprof! For help, type 'help'. > (pprof) top > Total: 4295 objects > 2150 50.1% 50.1% 2150 50.1% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:186 > 2141 49.8% 99.9% 2141 49.8% copy_string /usr/src/bro-2.2/src/util.cc:155 > 2 0.0% 100.0% 2 0.0% re_alloc /usr/src/bro-2.2/build/src/re-scan.cc:2287 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:110 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:133 > 0 0.0% 100.0% 2141 49.8% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:195 > 0 0.0% 100.0% 4 0.1% Connection::NextPacket /usr/src/bro-2.2/src/Conn.cc:259 > 0 0.0% 100.0% 4 0.1% NetSessions::DispatchPacket /usr/src/bro-2.2/src/Sessions.cc:189 > 0 0.0% 100.0% 4 0.1% NetSessions::DoNextPacket /usr/src/bro-2.2/src/Sessions.cc:709 > 0 0.0% 100.0% 4 0.1% NetSessions::NextPacket /usr/src/bro-2.2/src/Sessions.cc:247 > }} -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From noreply at bro.org Thu Dec 5 00:00:12 2013 From: noreply at bro.org (Merge Tracker) Date: Thu, 5 Dec 2013 00:00:12 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201312050800.rB580Cjv027794@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ------------ -------------- ---------- ---------- ------------- ---------- --------------------------------------- BIT-1101 [1] Bro Bernhard Amann - 2013-12-04 2.3 Normal Merge topic/bernhard/ssl_ciphers_vector BIT-1100 [2] Bro,Broccoli Jon Siwek - 2013-12-03 2.3 Normal topic/jsiwek/broccoli-vectors [3] BIT-1098 [4] Bro,Broccoli Jon Siwek - 2013-12-04 2.3 Normal topic/jsiwek/broxygen [5] BIT-1097 [6] Bro Robin Sommer - 2013-12-04 - Normal Unexpected string indexing behavior Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- --------- ---------- -------------------------------------------------- 2ea6011 [7] bro Jon Siwek 2013-12-03 Improve a unit test involving 'when' conditionals. [1] BIT-1101 https://bro-tracker.atlassian.net/browse/BIT-1101 [2] BIT-1100 https://bro-tracker.atlassian.net/browse/BIT-1100 [3] broccoli-vectors https://github.com/bro/bro,broccoli/tree/topic/jsiwek/broccoli-vectors [4] BIT-1098 https://bro-tracker.atlassian.net/browse/BIT-1098 [5] broxygen https://github.com/bro/bro,broccoli/tree/topic/jsiwek/broxygen [6] BIT-1097 https://bro-tracker.atlassian.net/browse/BIT-1097 [7] 2ea6011 https://github.com/bro/bro/commit/2ea6011186e91dff37c639667b1604d837363908 From jira at bro-tracker.atlassian.net Thu Dec 5 05:31:45 2013 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 5 Dec 2013 07:31:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1103) Memory leak in Bro Intel framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14908#comment-14908 ] Seth Hall commented on BIT-1103: -------------------------------- I have confirmed this leak, it seems to be an issue with the input framework. We'll be fixing it ASAP. Thanks for tracking it down. > Memory leak in Bro Intel framework > ---------------------------------- > > Key: BIT-1103 > URL: https://bro-tracker.atlassian.net/browse/BIT-1103 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Environment: Red Hat Enterprise Linux Server release 6.5 > Reporter: Andrew Hoying > Priority: High > Labels: intel, leak > > The policy/frameworks/intel/seen bro scripts have a memory leak. On my moderately busy Bro installation I am leaking about a gig of memory a day per worker process with the Intel framework enabled. I can replicate by adding the following to the local.bro default script and then running through a small PCAP with primarily dns, dhcp and syslog traffic. > {{ > @load policy/frameworks/intel/seen > redef Intel::read_files += { > "/usr/local/bro/spool/domain_suspicious.txt", > }; > }} > The intel file is in the following format, here's a few sample lines. It is generated automatically by CIF: > {{ > #fields indicator indicator_type meta.source meta.desc meta.url meta.cif_impact meta.cif_severity meta.cif_confidence > mete-tools.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=mete-tools.biz (public) - - 95 > rttvxygkmwlqmq.net Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=rttvxygkmwlqmq.net (public) - - 95 > podserveruho.com Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=podserveruho.com (public) - - 95 > wwfcogdgntlxw.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=wwfcogdgntlxw.biz (public) - - 95 > }} > I compiled bro with gperftool debug support and followed the instructions here: http://www.bro.org/development/howtos/leaks.html. (Note, the instructions are wrong on the flags for ./configure, you need to add --enable-perftools-debug to get the -m option for bro) > Here's the output from pprof top after running a PCAP trace with 10,000 packets. Running traces with more packets show a greater number of lost objects in the same code locations. > {{ > # pprof bin/bro "/tmp/bro.24541.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 > Using local file bin/bro. > Using local file /tmp/bro.24541.net_run-end.heap. > Welcome to pprof! For help, type 'help'. > (pprof) top > Total: 4295 objects > 2150 50.1% 50.1% 2150 50.1% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:186 > 2141 49.8% 99.9% 2141 49.8% copy_string /usr/src/bro-2.2/src/util.cc:155 > 2 0.0% 100.0% 2 0.0% re_alloc /usr/src/bro-2.2/build/src/re-scan.cc:2287 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:110 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:133 > 0 0.0% 100.0% 2141 49.8% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:195 > 0 0.0% 100.0% 4 0.1% Connection::NextPacket /usr/src/bro-2.2/src/Conn.cc:259 > 0 0.0% 100.0% 4 0.1% NetSessions::DispatchPacket /usr/src/bro-2.2/src/Sessions.cc:189 > 0 0.0% 100.0% 4 0.1% NetSessions::DoNextPacket /usr/src/bro-2.2/src/Sessions.cc:709 > 0 0.0% 100.0% 4 0.1% NetSessions::NextPacket /usr/src/bro-2.2/src/Sessions.cc:247 > }} -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Thu Dec 5 05:31:45 2013 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 5 Dec 2013 07:31:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1103) Memory leak in Bro Intel framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1103: --------------------------- Status: In Progress (was: Open) > Memory leak in Bro Intel framework > ---------------------------------- > > Key: BIT-1103 > URL: https://bro-tracker.atlassian.net/browse/BIT-1103 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Environment: Red Hat Enterprise Linux Server release 6.5 > Reporter: Andrew Hoying > Priority: High > Labels: intel, leak > > The policy/frameworks/intel/seen bro scripts have a memory leak. On my moderately busy Bro installation I am leaking about a gig of memory a day per worker process with the Intel framework enabled. I can replicate by adding the following to the local.bro default script and then running through a small PCAP with primarily dns, dhcp and syslog traffic. > {{ > @load policy/frameworks/intel/seen > redef Intel::read_files += { > "/usr/local/bro/spool/domain_suspicious.txt", > }; > }} > The intel file is in the following format, here's a few sample lines. It is generated automatically by CIF: > {{ > #fields indicator indicator_type meta.source meta.desc meta.url meta.cif_impact meta.cif_severity meta.cif_confidence > mete-tools.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=mete-tools.biz (public) - - 95 > rttvxygkmwlqmq.net Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=rttvxygkmwlqmq.net (public) - - 95 > podserveruho.com Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=podserveruho.com (public) - - 95 > wwfcogdgntlxw.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=wwfcogdgntlxw.biz (public) - - 95 > }} > I compiled bro with gperftool debug support and followed the instructions here: http://www.bro.org/development/howtos/leaks.html. (Note, the instructions are wrong on the flags for ./configure, you need to add --enable-perftools-debug to get the -m option for bro) > Here's the output from pprof top after running a PCAP trace with 10,000 packets. Running traces with more packets show a greater number of lost objects in the same code locations. > {{ > # pprof bin/bro "/tmp/bro.24541.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 > Using local file bin/bro. > Using local file /tmp/bro.24541.net_run-end.heap. > Welcome to pprof! For help, type 'help'. > (pprof) top > Total: 4295 objects > 2150 50.1% 50.1% 2150 50.1% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:186 > 2141 49.8% 99.9% 2141 49.8% copy_string /usr/src/bro-2.2/src/util.cc:155 > 2 0.0% 100.0% 2 0.0% re_alloc /usr/src/bro-2.2/build/src/re-scan.cc:2287 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:110 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:133 > 0 0.0% 100.0% 2141 49.8% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:195 > 0 0.0% 100.0% 4 0.1% Connection::NextPacket /usr/src/bro-2.2/src/Conn.cc:259 > 0 0.0% 100.0% 4 0.1% NetSessions::DispatchPacket /usr/src/bro-2.2/src/Sessions.cc:189 > 0 0.0% 100.0% 4 0.1% NetSessions::DoNextPacket /usr/src/bro-2.2/src/Sessions.cc:709 > 0 0.0% 100.0% 4 0.1% NetSessions::NextPacket /usr/src/bro-2.2/src/Sessions.cc:247 > }} -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Thu Dec 5 05:33:45 2013 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 5 Dec 2013 07:33:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1103) Memory leak in Bro Intel framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall reassigned BIT-1103: ------------------------------ Assignee: Bernhard Amann > Memory leak in Bro Intel framework > ---------------------------------- > > Key: BIT-1103 > URL: https://bro-tracker.atlassian.net/browse/BIT-1103 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Environment: Red Hat Enterprise Linux Server release 6.5 > Reporter: Andrew Hoying > Assignee: Bernhard Amann > Priority: High > Labels: intel, leak > > The policy/frameworks/intel/seen bro scripts have a memory leak. On my moderately busy Bro installation I am leaking about a gig of memory a day per worker process with the Intel framework enabled. I can replicate by adding the following to the local.bro default script and then running through a small PCAP with primarily dns, dhcp and syslog traffic. > {{ > @load policy/frameworks/intel/seen > redef Intel::read_files += { > "/usr/local/bro/spool/domain_suspicious.txt", > }; > }} > The intel file is in the following format, here's a few sample lines. It is generated automatically by CIF: > {{ > #fields indicator indicator_type meta.source meta.desc meta.url meta.cif_impact meta.cif_severity meta.cif_confidence > mete-tools.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=mete-tools.biz (public) - - 95 > rttvxygkmwlqmq.net Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=rttvxygkmwlqmq.net (public) - - 95 > podserveruho.com Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=podserveruho.com (public) - - 95 > wwfcogdgntlxw.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=wwfcogdgntlxw.biz (public) - - 95 > }} > I compiled bro with gperftool debug support and followed the instructions here: http://www.bro.org/development/howtos/leaks.html. (Note, the instructions are wrong on the flags for ./configure, you need to add --enable-perftools-debug to get the -m option for bro) > Here's the output from pprof top after running a PCAP trace with 10,000 packets. Running traces with more packets show a greater number of lost objects in the same code locations. > {{ > # pprof bin/bro "/tmp/bro.24541.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 > Using local file bin/bro. > Using local file /tmp/bro.24541.net_run-end.heap. > Welcome to pprof! For help, type 'help'. > (pprof) top > Total: 4295 objects > 2150 50.1% 50.1% 2150 50.1% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:186 > 2141 49.8% 99.9% 2141 49.8% copy_string /usr/src/bro-2.2/src/util.cc:155 > 2 0.0% 100.0% 2 0.0% re_alloc /usr/src/bro-2.2/build/src/re-scan.cc:2287 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:110 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:133 > 0 0.0% 100.0% 2141 49.8% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:195 > 0 0.0% 100.0% 4 0.1% Connection::NextPacket /usr/src/bro-2.2/src/Conn.cc:259 > 0 0.0% 100.0% 4 0.1% NetSessions::DispatchPacket /usr/src/bro-2.2/src/Sessions.cc:189 > 0 0.0% 100.0% 4 0.1% NetSessions::DoNextPacket /usr/src/bro-2.2/src/Sessions.cc:709 > 0 0.0% 100.0% 4 0.1% NetSessions::NextPacket /usr/src/bro-2.2/src/Sessions.cc:247 > }} -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Thu Dec 5 05:51:45 2013 From: jira at bro-tracker.atlassian.net (Michael Stone (JIRA)) Date: Thu, 5 Dec 2013 07:51:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1104) Add tracking for MSIE 11 In-Reply-To: References: Message-ID: Michael Stone created BIT-1104: ---------------------------------- Summary: Add tracking for MSIE 11 Key: BIT-1104 URL: https://bro-tracker.atlassian.net/browse/BIT-1104 Project: Bro Issue Tracker Issue Type: Patch Components: Bro Affects Versions: 2.1 Environment: Ubuntu Reporter: Michael Stone MSIE 11.0 currently shows up as . It looks like MS might have changed it's user agent string and doesn't include "MSIE". I added the following to /usr/local/bro/share/bro/base/frameworks/software/main.bro just below the "MSIE" block and above the "Safari" block. else if ( /Trident\/7.0/ in uparsed_version ) { if ( /rv:11\.0/ in unparsed_version ) { software_name = "MSIE"; v = [$major=11,$minor=0]; } } Disclaimer: I'm fairly new to working with Bro so this might not be the best way, but it seems to be working for me. Thanks! -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Thu Dec 5 07:02:45 2013 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Thu, 5 Dec 2013 09:02:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1103) Memory leak in Bro Intel framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14909#comment-14909 ] Bernhard Amann commented on BIT-1103: ------------------------------------- Seth and me tracked down the root of the leak - the input manager uses the length of the root-record while deleting values instead of the length of the unrolled record. I will fix this in a few hours (well, after re-checking it really fixes it), thank you very much for the very thorough bug-report. > Memory leak in Bro Intel framework > ---------------------------------- > > Key: BIT-1103 > URL: https://bro-tracker.atlassian.net/browse/BIT-1103 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Environment: Red Hat Enterprise Linux Server release 6.5 > Reporter: Andrew Hoying > Assignee: Bernhard Amann > Priority: High > Labels: intel, leak > > The policy/frameworks/intel/seen bro scripts have a memory leak. On my moderately busy Bro installation I am leaking about a gig of memory a day per worker process with the Intel framework enabled. I can replicate by adding the following to the local.bro default script and then running through a small PCAP with primarily dns, dhcp and syslog traffic. > {{ > @load policy/frameworks/intel/seen > redef Intel::read_files += { > "/usr/local/bro/spool/domain_suspicious.txt", > }; > }} > The intel file is in the following format, here's a few sample lines. It is generated automatically by CIF: > {{ > #fields indicator indicator_type meta.source meta.desc meta.url meta.cif_impact meta.cif_severity meta.cif_confidence > mete-tools.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=mete-tools.biz (public) - - 95 > rttvxygkmwlqmq.net Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=rttvxygkmwlqmq.net (public) - - 95 > podserveruho.com Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=podserveruho.com (public) - - 95 > wwfcogdgntlxw.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=wwfcogdgntlxw.biz (public) - - 95 > }} > I compiled bro with gperftool debug support and followed the instructions here: http://www.bro.org/development/howtos/leaks.html. (Note, the instructions are wrong on the flags for ./configure, you need to add --enable-perftools-debug to get the -m option for bro) > Here's the output from pprof top after running a PCAP trace with 10,000 packets. Running traces with more packets show a greater number of lost objects in the same code locations. > {{ > # pprof bin/bro "/tmp/bro.24541.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 > Using local file bin/bro. > Using local file /tmp/bro.24541.net_run-end.heap. > Welcome to pprof! For help, type 'help'. > (pprof) top > Total: 4295 objects > 2150 50.1% 50.1% 2150 50.1% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:186 > 2141 49.8% 99.9% 2141 49.8% copy_string /usr/src/bro-2.2/src/util.cc:155 > 2 0.0% 100.0% 2 0.0% re_alloc /usr/src/bro-2.2/build/src/re-scan.cc:2287 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:110 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:133 > 0 0.0% 100.0% 2141 49.8% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:195 > 0 0.0% 100.0% 4 0.1% Connection::NextPacket /usr/src/bro-2.2/src/Conn.cc:259 > 0 0.0% 100.0% 4 0.1% NetSessions::DispatchPacket /usr/src/bro-2.2/src/Sessions.cc:189 > 0 0.0% 100.0% 4 0.1% NetSessions::DoNextPacket /usr/src/bro-2.2/src/Sessions.cc:709 > 0 0.0% 100.0% 4 0.1% NetSessions::NextPacket /usr/src/bro-2.2/src/Sessions.cc:247 > }} -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Thu Dec 5 07:44:45 2013 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 5 Dec 2013 09:44:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1102) Logs use tab delimiter yet do not remove tabs from logged data In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1102: --------------------------- Status: In Progress (was: Open) > Logs use tab delimiter yet do not remove tabs from logged data > -------------------------------------------------------------- > > Key: BIT-1102 > URL: https://bro-tracker.atlassian.net/browse/BIT-1102 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Environment: ubuntu 12.04LTS server > Reporter: April Lorenzen > Labels: http.log > Attachments: 4872c1c6d6983c4b6cf4e0f83baf5ec5.pcap > > > The attached pcap processed with bro -r pcapname will demonstrate the issue. View the http.log after processing and note that the field delimiter appears within a field. > I tried searching the official github mirror but don't know bro well enough to find the spot to look. I'd be happy to work on fixing this if you point me to the problem area (that's the hard part, right?) -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Thu Dec 5 07:46:45 2013 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 5 Dec 2013 09:46:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1102) Logs use tab delimiter yet do not remove tabs from logged data In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall reassigned BIT-1102: ------------------------------ Assignee: Seth Hall > Logs use tab delimiter yet do not remove tabs from logged data > -------------------------------------------------------------- > > Key: BIT-1102 > URL: https://bro-tracker.atlassian.net/browse/BIT-1102 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Environment: ubuntu 12.04LTS server > Reporter: April Lorenzen > Assignee: Seth Hall > Labels: http.log > Attachments: 4872c1c6d6983c4b6cf4e0f83baf5ec5.pcap > > > The attached pcap processed with bro -r pcapname will demonstrate the issue. View the http.log after processing and note that the field delimiter appears within a field. > I tried searching the official github mirror but don't know bro well enough to find the spot to look. I'd be happy to work on fixing this if you point me to the problem area (that's the hard part, right?) -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Thu Dec 5 07:46:45 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 5 Dec 2013 09:46:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1101) Merge topic/bernhard/ssl_ciphers_vector In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1101: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Merge topic/bernhard/ssl_ciphers_vector > --------------------------------------- > > Key: BIT-1101 > URL: https://bro-tracker.atlassian.net/browse/BIT-1101 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Fix For: 2.3 > > > topic/bernhard/ssl_ciphers_vector changes ciphers in the ssl_client_hello from a set into a vector. This preserves the ordering of the cipher suites the client sent, allowing e.g. better client fingerprinting. -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Thu Dec 5 07:46:45 2013 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 5 Dec 2013 09:46:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1102) Logs use tab delimiter yet do not remove tabs from logged data In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14910#comment-14910 ] Seth Hall commented on BIT-1102: -------------------------------- Which line in http.log are you specifically referring to? I don't see any problems with this log. We specifically handle the case of field separators within fields by hex escaping. > Logs use tab delimiter yet do not remove tabs from logged data > -------------------------------------------------------------- > > Key: BIT-1102 > URL: https://bro-tracker.atlassian.net/browse/BIT-1102 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Environment: ubuntu 12.04LTS server > Reporter: April Lorenzen > Labels: http.log > Attachments: 4872c1c6d6983c4b6cf4e0f83baf5ec5.pcap > > > The attached pcap processed with bro -r pcapname will demonstrate the issue. View the http.log after processing and note that the field delimiter appears within a field. > I tried searching the official github mirror but don't know bro well enough to find the spot to look. I'd be happy to work on fixing this if you point me to the problem area (that's the hard part, right?) -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Thu Dec 5 07:48:45 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 5 Dec 2013 09:48:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1100) topic/jsiwek/broccoli-vectors In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1100: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > topic/jsiwek/broccoli-vectors > ----------------------------- > > Key: BIT-1100 > URL: https://bro-tracker.atlassian.net/browse/BIT-1100 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro, Broccoli > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > This branch is in the bro and broccoli repos and adds support for broccoli clients to receive events that have arguments w/ vector values. > Sending events that have arguments w/ vector values is still unsupported. (Broccoli generally seems to be limited in the complexity of types it can create compared to Bro). -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Thu Dec 5 07:50:45 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 5 Dec 2013 09:50:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1097) Unexpected string indexing behavior In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1097: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Unexpected string indexing behavior > ----------------------------------- > > Key: BIT-1097 > URL: https://bro-tracker.atlassian.net/browse/BIT-1097 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Robin Sommer > > Playing with string indexing/slicing, I'm seeing some (I think) non-intuitive behavior: > {code} > global s = "012345"; > print "A"; > print s[1:-1]; > print s[1:-2]; > print s[1:-3]; > print s[1:-4]; > print s[1:-5]; > print s[1:-6]; > print s[1:-7]; > print s[1:-8]; > print s[1:-9]; > print ""; > print "B"; > print s[-1:-1]; > print s[-1:-2]; > print s[-1:-3]; > print s[-1:-4]; > {code} > This prints: > {code} > A > 12345 > 1234 > 123 > 12 > 1 > 12345 > 12345 > 12345 > B > 5 > 5 > 5 > {code} > I would instead have expected: > (1) A to print empty lines for all cases with the 2nd index <= -6? > (2) B to print empty lines for all cases with the 2nd index <= -2? > So, is this intentional? -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Thu Dec 5 07:52:45 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 5 Dec 2013 09:52:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1098) topic/jsiwek/broxygen In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1098: ------------------------------ Status: Open (was: Merge Request) > topic/jsiwek/broxygen > --------------------- > > Key: BIT-1098 > URL: https://bro-tracker.atlassian.net/browse/BIT-1098 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, Broccoli > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > This branch is in the bro and broccoli repos and improves the automated script-reference documentation generation process, broxygen. > This should address issues in BIT-701 and BIT-751, let me know if there's anything not covered that's still relevant. > Highlights: > - Remove {{--doc-scripts}} and {{-Z}} options to toggle documentation mode -- the parser is now always instrumented to gather documentation from comments of the form "##", "##!", or "##<". > - Raw comments are available at runtime through several BIF functions: {{get_*_comments}}; > - Add {{--broxygen}} and {{-X}} options to toggle generating reST-format documentation output, driven by a config file argument. > - Add a "broxygen" Sphinx extension domain, allowing certain pieces of documentation to be generated on-the-fly via invoking a Bro process. Re-organized/cleaned up the Sphinx source tree in {{doc/}} to use this in some places. -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From randy at packetchaser.org Thu Dec 5 09:54:06 2013 From: randy at packetchaser.org (Randy Caldejon) Date: Thu, 5 Dec 2013 12:54:06 -0500 Subject: [Bro-Dev] Broccoli and vectors Message-ID: <4C6EB2F1-ABB1-40CF-AB6B-D13A37445D79@packetchaser.org> Hi, I'm implementing an application that sends DNS::Info records via Broccoli to Bro. However, it appears that Broccoli does not fully support vectors. Is this correct? If it does, can somebody point me to an example on how to populate a vector using the Broccoli C API. I searched through the Broccoli docs but could not find anything. Thanks, -- Randy From robin at icir.org Thu Dec 5 09:58:58 2013 From: robin at icir.org (Robin Sommer) Date: Thu, 5 Dec 2013 09:58:58 -0800 Subject: [Bro-Dev] Broccoli and vectors In-Reply-To: <4C6EB2F1-ABB1-40CF-AB6B-D13A37445D79@packetchaser.org> References: <4C6EB2F1-ABB1-40CF-AB6B-D13A37445D79@packetchaser.org> Message-ID: <20131205175858.GA85508@icir.org> Half of the support for vectors is there since yesterday: https://github.com/bro/broccoli/commit/756a8a733b1f03b94afcbb93807813a89b3cfb89 However it sounds like you need the opposite: there's no support yet for producing events with vectors. Robin On Thu, Dec 05, 2013 at 12:54 -0500, you wrote: > Hi, > > I'm implementing an application that sends DNS::Info records via > Broccoli to Bro. However, it appears that Broccoli does not fully > support vectors. Is this correct? If it does, can somebody point me > to an example on how to populate a vector using the Broccoli C API. I > searched through the Broccoli docs but could not find anything. > > Thanks, > > -- Randy > > > > > > > > > > > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From randy at packetchaser.org Thu Dec 5 10:04:52 2013 From: randy at packetchaser.org (Randy Caldejon) Date: Thu, 5 Dec 2013 13:04:52 -0500 Subject: [Bro-Dev] Broccoli and vectors In-Reply-To: <20131205175858.GA85508@icir.org> References: <4C6EB2F1-ABB1-40CF-AB6B-D13A37445D79@packetchaser.org> <20131205175858.GA85508@icir.org> Message-ID: Ok, thanks for the quick response. --Randy > On Dec 5, 2013, at 12:58, Robin Sommer wrote: > > Half of the support for vectors is there since yesterday: > > https://github.com/bro/broccoli/commit/756a8a733b1f03b94afcbb93807813a89b3cfb89 > > However it sounds like you need the opposite: there's no support yet > for producing events with vectors. > > Robin > >> On Thu, Dec 05, 2013 at 12:54 -0500, you wrote: >> >> Hi, >> >> I'm implementing an application that sends DNS::Info records via >> Broccoli to Bro. However, it appears that Broccoli does not fully >> support vectors. Is this correct? If it does, can somebody point me >> to an example on how to populate a vector using the Broccoli C API. I >> searched through the Broccoli docs but could not find anything. >> >> Thanks, >> >> -- Randy >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> bro-dev mailing list >> bro-dev at bro.org >> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > > > -- > Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org > ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From jsiwek at illinois.edu Thu Dec 5 10:11:17 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Thu, 5 Dec 2013 18:11:17 +0000 Subject: [Bro-Dev] Broccoli and vectors In-Reply-To: <4C6EB2F1-ABB1-40CF-AB6B-D13A37445D79@packetchaser.org> References: <4C6EB2F1-ABB1-40CF-AB6B-D13A37445D79@packetchaser.org> Message-ID: <7ECAC7C4-2217-449B-9E26-96E7E73A2F97@illinois.edu> On Dec 5, 2013, at 11:54 AM, Randy Caldejon wrote: > I'm implementing an application that sends DNS::Info records via Broccoli to Bro. However, it appears that Broccoli does not fully support vectors. Is this correct? If it does, can somebody point me to an example on how to populate a vector using the Broccoli C API. I searched through the Broccoli docs but could not find anything. You?re right, Broccoli does not support sending vectors (though I added support for recv?ing them earlier this week). A possible workaround for sending vectors via Broccoli may be to send each element individually in an event and have Bro construct the vector on the other side one element at a time. - Jon From jira at bro-tracker.atlassian.net Thu Dec 5 11:01:45 2013 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Thu, 5 Dec 2013 13:01:45 -0600 (CST) Subject: [Bro-Dev] ***SPAM*** [JIRA] (BIT-1103) Memory leak in Bro Intel framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1103: -------------------------------- Status: Merge Request (was: In Progress) > Memory leak in Bro Intel framework > ---------------------------------- > > Key: BIT-1103 > URL: https://bro-tracker.atlassian.net/browse/BIT-1103 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Environment: Red Hat Enterprise Linux Server release 6.5 > Reporter: Andrew Hoying > Assignee: Bernhard Amann > Priority: High > Labels: intel, leak > > The policy/frameworks/intel/seen bro scripts have a memory leak. On my moderately busy Bro installation I am leaking about a gig of memory a day per worker process with the Intel framework enabled. I can replicate by adding the following to the local.bro default script and then running through a small PCAP with primarily dns, dhcp and syslog traffic. > {{ > @load policy/frameworks/intel/seen > redef Intel::read_files += { > "/usr/local/bro/spool/domain_suspicious.txt", > }; > }} > The intel file is in the following format, here's a few sample lines. It is generated automatically by CIF: > {{ > #fields indicator indicator_type meta.source meta.desc meta.url meta.cif_impact meta.cif_severity meta.cif_confidence > mete-tools.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=mete-tools.biz (public) - - 95 > rttvxygkmwlqmq.net Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=rttvxygkmwlqmq.net (public) - - 95 > podserveruho.com Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=podserveruho.com (public) - - 95 > wwfcogdgntlxw.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=wwfcogdgntlxw.biz (public) - - 95 > }} > I compiled bro with gperftool debug support and followed the instructions here: http://www.bro.org/development/howtos/leaks.html. (Note, the instructions are wrong on the flags for ./configure, you need to add --enable-perftools-debug to get the -m option for bro) > Here's the output from pprof top after running a PCAP trace with 10,000 packets. Running traces with more packets show a greater number of lost objects in the same code locations. > {{ > # pprof bin/bro "/tmp/bro.24541.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 > Using local file bin/bro. > Using local file /tmp/bro.24541.net_run-end.heap. > Welcome to pprof! For help, type 'help'. > (pprof) top > Total: 4295 objects > 2150 50.1% 50.1% 2150 50.1% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:186 > 2141 49.8% 99.9% 2141 49.8% copy_string /usr/src/bro-2.2/src/util.cc:155 > 2 0.0% 100.0% 2 0.0% re_alloc /usr/src/bro-2.2/build/src/re-scan.cc:2287 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:110 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:133 > 0 0.0% 100.0% 2141 49.8% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:195 > 0 0.0% 100.0% 4 0.1% Connection::NextPacket /usr/src/bro-2.2/src/Conn.cc:259 > 0 0.0% 100.0% 4 0.1% NetSessions::DispatchPacket /usr/src/bro-2.2/src/Sessions.cc:189 > 0 0.0% 100.0% 4 0.1% NetSessions::DoNextPacket /usr/src/bro-2.2/src/Sessions.cc:709 > 0 0.0% 100.0% 4 0.1% NetSessions::NextPacket /usr/src/bro-2.2/src/Sessions.cc:247 > }} -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Thu Dec 5 11:01:45 2013 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Thu, 5 Dec 2013 13:01:45 -0600 (CST) Subject: [Bro-Dev] ***SPAM*** [JIRA] (BIT-1103) Memory leak in Bro Intel framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14911#comment-14911 ] Bernhard Amann commented on BIT-1103: ------------------------------------- verified patch is in topic/bernhard/ticket1103 > Memory leak in Bro Intel framework > ---------------------------------- > > Key: BIT-1103 > URL: https://bro-tracker.atlassian.net/browse/BIT-1103 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Environment: Red Hat Enterprise Linux Server release 6.5 > Reporter: Andrew Hoying > Assignee: Bernhard Amann > Priority: High > Labels: intel, leak > > The policy/frameworks/intel/seen bro scripts have a memory leak. On my moderately busy Bro installation I am leaking about a gig of memory a day per worker process with the Intel framework enabled. I can replicate by adding the following to the local.bro default script and then running through a small PCAP with primarily dns, dhcp and syslog traffic. > {{ > @load policy/frameworks/intel/seen > redef Intel::read_files += { > "/usr/local/bro/spool/domain_suspicious.txt", > }; > }} > The intel file is in the following format, here's a few sample lines. It is generated automatically by CIF: > {{ > #fields indicator indicator_type meta.source meta.desc meta.url meta.cif_impact meta.cif_severity meta.cif_confidence > mete-tools.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=mete-tools.biz (public) - - 95 > rttvxygkmwlqmq.net Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=rttvxygkmwlqmq.net (public) - - 95 > podserveruho.com Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=podserveruho.com (public) - - 95 > wwfcogdgntlxw.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=wwfcogdgntlxw.biz (public) - - 95 > }} > I compiled bro with gperftool debug support and followed the instructions here: http://www.bro.org/development/howtos/leaks.html. (Note, the instructions are wrong on the flags for ./configure, you need to add --enable-perftools-debug to get the -m option for bro) > Here's the output from pprof top after running a PCAP trace with 10,000 packets. Running traces with more packets show a greater number of lost objects in the same code locations. > {{ > # pprof bin/bro "/tmp/bro.24541.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 > Using local file bin/bro. > Using local file /tmp/bro.24541.net_run-end.heap. > Welcome to pprof! For help, type 'help'. > (pprof) top > Total: 4295 objects > 2150 50.1% 50.1% 2150 50.1% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:186 > 2141 49.8% 99.9% 2141 49.8% copy_string /usr/src/bro-2.2/src/util.cc:155 > 2 0.0% 100.0% 2 0.0% re_alloc /usr/src/bro-2.2/build/src/re-scan.cc:2287 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:110 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:133 > 0 0.0% 100.0% 2141 49.8% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:195 > 0 0.0% 100.0% 4 0.1% Connection::NextPacket /usr/src/bro-2.2/src/Conn.cc:259 > 0 0.0% 100.0% 4 0.1% NetSessions::DispatchPacket /usr/src/bro-2.2/src/Sessions.cc:189 > 0 0.0% 100.0% 4 0.1% NetSessions::DoNextPacket /usr/src/bro-2.2/src/Sessions.cc:709 > 0 0.0% 100.0% 4 0.1% NetSessions::NextPacket /usr/src/bro-2.2/src/Sessions.cc:247 > }} -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From randy at packetchaser.org Thu Dec 5 11:22:48 2013 From: randy at packetchaser.org (Randy Caldejon) Date: Thu, 5 Dec 2013 14:22:48 -0500 Subject: [Bro-Dev] Broccoli and vectors In-Reply-To: <7ECAC7C4-2217-449B-9E26-96E7E73A2F97@illinois.edu> References: <4C6EB2F1-ABB1-40CF-AB6B-D13A37445D79@packetchaser.org> <7ECAC7C4-2217-449B-9E26-96E7E73A2F97@illinois.edu> Message-ID: <4D338046-7412-4529-B15C-6EA9B213F0C3@packetchaser.org> Thanks. I'll try that. --Randy > On Dec 5, 2013, at 13:11, "Siwek, Jonathan Luke" wrote: > > >> On Dec 5, 2013, at 11:54 AM, Randy Caldejon wrote: >> >> I'm implementing an application that sends DNS::Info records via Broccoli to Bro. However, it appears that Broccoli does not fully support vectors. Is this correct? If it does, can somebody point me to an example on how to populate a vector using the Broccoli C API. I searched through the Broccoli docs but could not find anything. > > You?re right, Broccoli does not support sending vectors (though I added support for recv?ing them earlier this week). > > A possible workaround for sending vectors via Broccoli may be to send each element individually in an event and have Bro construct the vector on the other side one element at a time. > > - Jon From jira at bro-tracker.atlassian.net Thu Dec 5 11:39:45 2013 From: jira at bro-tracker.atlassian.net (Andrew Hoying (JIRA)) Date: Thu, 5 Dec 2013 13:39:45 -0600 (CST) Subject: [Bro-Dev] ***SPAM*** [JIRA] (BIT-1103) Memory leak in Bro Intel framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14912#comment-14912 ] Andrew Hoying commented on BIT-1103: ------------------------------------ I applied the patch against 2.2 and verified that it fixed the memory leak on my system. Thank you! > Memory leak in Bro Intel framework > ---------------------------------- > > Key: BIT-1103 > URL: https://bro-tracker.atlassian.net/browse/BIT-1103 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Environment: Red Hat Enterprise Linux Server release 6.5 > Reporter: Andrew Hoying > Assignee: Bernhard Amann > Priority: High > Labels: intel, leak > > The policy/frameworks/intel/seen bro scripts have a memory leak. On my moderately busy Bro installation I am leaking about a gig of memory a day per worker process with the Intel framework enabled. I can replicate by adding the following to the local.bro default script and then running through a small PCAP with primarily dns, dhcp and syslog traffic. > {{ > @load policy/frameworks/intel/seen > redef Intel::read_files += { > "/usr/local/bro/spool/domain_suspicious.txt", > }; > }} > The intel file is in the following format, here's a few sample lines. It is generated automatically by CIF: > {{ > #fields indicator indicator_type meta.source meta.desc meta.url meta.cif_impact meta.cif_severity meta.cif_confidence > mete-tools.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=mete-tools.biz (public) - - 95 > rttvxygkmwlqmq.net Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=rttvxygkmwlqmq.net (public) - - 95 > podserveruho.com Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=podserveruho.com (public) - - 95 > wwfcogdgntlxw.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=wwfcogdgntlxw.biz (public) - - 95 > }} > I compiled bro with gperftool debug support and followed the instructions here: http://www.bro.org/development/howtos/leaks.html. (Note, the instructions are wrong on the flags for ./configure, you need to add --enable-perftools-debug to get the -m option for bro) > Here's the output from pprof top after running a PCAP trace with 10,000 packets. Running traces with more packets show a greater number of lost objects in the same code locations. > {{ > # pprof bin/bro "/tmp/bro.24541.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 > Using local file bin/bro. > Using local file /tmp/bro.24541.net_run-end.heap. > Welcome to pprof! For help, type 'help'. > (pprof) top > Total: 4295 objects > 2150 50.1% 50.1% 2150 50.1% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:186 > 2141 49.8% 99.9% 2141 49.8% copy_string /usr/src/bro-2.2/src/util.cc:155 > 2 0.0% 100.0% 2 0.0% re_alloc /usr/src/bro-2.2/build/src/re-scan.cc:2287 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:110 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:133 > 0 0.0% 100.0% 2141 49.8% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:195 > 0 0.0% 100.0% 4 0.1% Connection::NextPacket /usr/src/bro-2.2/src/Conn.cc:259 > 0 0.0% 100.0% 4 0.1% NetSessions::DispatchPacket /usr/src/bro-2.2/src/Sessions.cc:189 > 0 0.0% 100.0% 4 0.1% NetSessions::DoNextPacket /usr/src/bro-2.2/src/Sessions.cc:709 > 0 0.0% 100.0% 4 0.1% NetSessions::NextPacket /usr/src/bro-2.2/src/Sessions.cc:247 > }} -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Thu Dec 5 12:07:45 2013 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 5 Dec 2013 14:07:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1104) Add tracking for MSIE 11 In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1104: --------------------------- Status: Merge Request (was: Open) > Add tracking for MSIE 11 > ------------------------ > > Key: BIT-1104 > URL: https://bro-tracker.atlassian.net/browse/BIT-1104 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.1 > Environment: Ubuntu > Reporter: Michael Stone > Labels: analyzer > > MSIE 11.0 currently shows up as . It looks like MS might have changed it's user agent string and doesn't include "MSIE". I added the following to /usr/local/bro/share/bro/base/frameworks/software/main.bro > just below the "MSIE" block and above the "Safari" block. > else if ( /Trident\/7.0/ in uparsed_version ) > { > if ( /rv:11\.0/ in unparsed_version ) { > software_name = "MSIE"; > v = [$major=11,$minor=0]; > } > } > Disclaimer: I'm fairly new to working with Bro so this might not be the best way, but it seems to be working for me. > Thanks! -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Thu Dec 5 12:07:45 2013 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 5 Dec 2013 14:07:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1104) Add tracking for MSIE 11 In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14913#comment-14913 ] Seth Hall commented on BIT-1104: -------------------------------- I have made some changes to your edit and it's in the topic/seth/ie11-software-parsing branch. Thanks. > Add tracking for MSIE 11 > ------------------------ > > Key: BIT-1104 > URL: https://bro-tracker.atlassian.net/browse/BIT-1104 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.1 > Environment: Ubuntu > Reporter: Michael Stone > Labels: analyzer > > MSIE 11.0 currently shows up as . It looks like MS might have changed it's user agent string and doesn't include "MSIE". I added the following to /usr/local/bro/share/bro/base/frameworks/software/main.bro > just below the "MSIE" block and above the "Safari" block. > else if ( /Trident\/7.0/ in uparsed_version ) > { > if ( /rv:11\.0/ in unparsed_version ) { > software_name = "MSIE"; > v = [$major=11,$minor=0]; > } > } > Disclaimer: I'm fairly new to working with Bro so this might not be the best way, but it seems to be working for me. > Thanks! -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Thu Dec 5 12:13:45 2013 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 5 Dec 2013 14:13:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1104) Add tracking for MSIE 11 In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall reassigned BIT-1104: ------------------------------ Assignee: Seth Hall > Add tracking for MSIE 11 > ------------------------ > > Key: BIT-1104 > URL: https://bro-tracker.atlassian.net/browse/BIT-1104 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.1 > Environment: Ubuntu > Reporter: Michael Stone > Assignee: Seth Hall > Labels: analyzer > > MSIE 11.0 currently shows up as . It looks like MS might have changed it's user agent string and doesn't include "MSIE". I added the following to /usr/local/bro/share/bro/base/frameworks/software/main.bro > just below the "MSIE" block and above the "Safari" block. > else if ( /Trident\/7.0/ in uparsed_version ) > { > if ( /rv:11\.0/ in unparsed_version ) { > software_name = "MSIE"; > v = [$major=11,$minor=0]; > } > } > Disclaimer: I'm fairly new to working with Bro so this might not be the best way, but it seems to be working for me. > Thanks! -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Thu Dec 5 13:02:45 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 5 Dec 2013 15:02:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1105) /topic/jsiwek/misc-fixes In-Reply-To: References: Message-ID: Jon Siwek created BIT-1105: ------------------------------ Summary: /topic/jsiwek/misc-fixes Key: BIT-1105 URL: https://bro-tracker.atlassian.net/browse/BIT-1105 Project: Bro Issue Tracker Issue Type: Problem Components: Bro, Broccoli, BroControl Affects Versions: git/master Reporter: Jon Siwek Priority: High Fix For: 2.3 This is in bro, broccoli, and broctl. It fixes various build/test/coverity failures. The ref counting fix may be a pre-existing issue relevant to 2.2, but just coincidentally exposed on one jenkins node now. -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Thu Dec 5 13:04:45 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 5 Dec 2013 15:04:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1105) /topic/jsiwek/misc-fixes In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1105: --------------------------- Status: Merge Request (was: Open) > /topic/jsiwek/misc-fixes > ------------------------ > > Key: BIT-1105 > URL: https://bro-tracker.atlassian.net/browse/BIT-1105 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro, Broccoli, BroControl > Affects Versions: git/master > Reporter: Jon Siwek > Priority: High > Fix For: 2.3 > > > This is in bro, broccoli, and broctl. It fixes various build/test/coverity failures. > The ref counting fix may be a pre-existing issue relevant to 2.2, but just coincidentally exposed on one jenkins node now. -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Thu Dec 5 16:01:45 2013 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Thu, 5 Dec 2013 18:01:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1106) Merge topic/bernhard/input-error-fixes In-Reply-To: References: Message-ID: Bernhard Amann created BIT-1106: ----------------------------------- Summary: Merge topic/bernhard/input-error-fixes Key: BIT-1106 URL: https://bro-tracker.atlassian.net/browse/BIT-1106 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: git/master Reporter: Bernhard Amann The branch topic/bernhard/input-error-fixes fixes a number of issues of the input framework that all have to do with errors: -First: Due to architectural constraints, it is very hard for the input framework to handle optional records. For an optional record, either the whole record has to be missing, or all non-optional elements of the record have to be defined. This information is not available to input readers after the records have been unrolled into the threading types. Behavior so far was to treat optional records like they are non-optional, without warning. The patch changes this behavior to emit an error on stream-creation (during type-checking) and refusing to open the file. I think this is a better idea - the behavior so far was undocumented and unintuitive. - Second: For table and event streams, reader backend creation was done very early, before actually checking if all arguments are valid. Initialization is moved after the checks now - this makes a number of delete statements unnecessary. Also - I suspect threads of failed input reader instances were not deleted until shutdown - Third: Add a couple more consistency checks, e.g. checking if the destination value of a table has the same type as we need. We did not check everything in all instances, instead we just assigned the things without caring (which works, but is not really desirable). This change also exposed a few bugs in other testcases where table definitions were wrong (did not respect $want_record) - Fourth: Improve error messages and write testcases for all error messages (I think). -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Thu Dec 5 16:01:45 2013 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Thu, 5 Dec 2013 18:01:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1106) Merge topic/bernhard/input-error-fixes In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1106: -------------------------------- Status: Merge Request (was: Open) > Merge topic/bernhard/input-error-fixes > -------------------------------------- > > Key: BIT-1106 > URL: https://bro-tracker.atlassian.net/browse/BIT-1106 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > > The branch topic/bernhard/input-error-fixes fixes a number of issues of the input framework that all have to do with errors: > -First: > Due to architectural constraints, it is very hard for the input framework to handle optional records. For an optional record, either the whole record has to be missing, or all non-optional elements of the record have to be defined. This information is not available to input readers after the records have been unrolled into the threading types. > Behavior so far was to treat optional records like they are non-optional, without warning. The patch changes this behavior to emit an error on stream-creation (during type-checking) and refusing to open the file. I think this is a better idea - the behavior so far was undocumented and unintuitive. > - Second: > For table and event streams, reader backend creation was done very early, before actually checking if all arguments are valid. Initialization is moved after the checks now - this makes a number of delete statements unnecessary. Also - I suspect threads of failed input reader instances were not deleted until shutdown > - Third: > Add a couple more consistency checks, e.g. checking if the destination value of a table has the same type as we need. We did not check everything in all instances, instead we just assigned the things without caring (which works, but is not really desirable). > This change also exposed a few bugs in other testcases where table definitions were wrong (did not respect $want_record) > - Fourth: > Improve error messages and write testcases for all error messages (I think). -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From noreply at bro.org Fri Dec 6 00:00:14 2013 From: noreply at bro.org (Merge Tracker) Date: Fri, 6 Dec 2013 00:00:14 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201312060800.rB680Eme011682@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------------------- -------------- -------------- ---------- ------------- ---------- -------------------------------------- BIT-1106 [1] Bro Bernhard Amann - 2013-12-05 - Normal Merge topic/bernhard/input-error-fixes BIT-1105 [2] Bro,Broccoli,BroControl Jon Siwek - 2013-12-05 2.3 High /topic/jsiwek/misc-fixes BIT-1104 [3] Bro Michael Stone Seth Hall 2013-12-05 - Normal Add tracking for MSIE 11 BIT-1103 [4] Bro Andrew Hoying Bernhard Amann 2013-12-05 - High Memory leak in Bro Intel framework [1] BIT-1106 https://bro-tracker.atlassian.net/browse/BIT-1106 [2] BIT-1105 https://bro-tracker.atlassian.net/browse/BIT-1105 [3] BIT-1104 https://bro-tracker.atlassian.net/browse/BIT-1104 [4] BIT-1103 https://bro-tracker.atlassian.net/browse/BIT-1103 From jira at bro-tracker.atlassian.net Fri Dec 6 07:53:45 2013 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Fri, 6 Dec 2013 09:53:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1107) Documentation of BIFs that take variable number of arguments In-Reply-To: References: Message-ID: Daniel Thayer created BIT-1107: ---------------------------------- Summary: Documentation of BIFs that take variable number of arguments Key: BIT-1107 URL: https://bro-tracker.atlassian.net/browse/BIT-1107 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Reporter: Daniel Thayer The function prototype for BIFs that take a variable number of arguments appears in an altered form in the online documentation. Here is a comparison of how these functions appear in the source code, versus what they look like in the online documentation: md5_hash%(...%) ===> Type : function (va_args: any) order%(v: any, ...%) ===> Type : function (va_args: any) sort%(v: any, ...%) ===> Type : function (va_args: any) cat_sep%(sep: string, def: string, ...%) ===> Type : function (va_args: any) The functions that have a named argument ("v" in sort, or "sep" in cat_sep) have those arguments described in the online documentation, but we cannot see them in the function prototype (only "va_args" is shown, which isn't actually the name of any function argument). -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Fri Dec 6 07:58:45 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 6 Dec 2013 09:58:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1098) topic/jsiwek/broxygen In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14914#comment-14914 ] Jon Siwek commented on BIT-1098: -------------------------------- {quote} - what would you think about moving the global type alias table into BroType as a private static member, along with a few static accessor methods to manipulate it? {quote} Seems better. Done. {quote} - if there's a bro binary in PATH, it seems that "make doc" picks up that one instead of the one in {{src/build/bro}} {quote} Fixed. {quote} - "make doc" shows {{received termination signal}} right before {{Running Sphinx}}. Is that expected? {quote} Expected, I guess. It's just a result of the broxygen/ package calling {{terminate()}} in {{bro_init()}}. I do that so that scripts like {{listen.bro}} can be loaded during the doc-generation process. I don't want to filter all stderr to /dev/null, because that can give useful diagnostics when something goes wrong. Could grep out just that string, but I'm kind of fine with it... {quote} -- I'd remove the "Param" in the function descriptions (or is that a Sphinx thing?) {quote} (I think) that was there in the previous Broxygen version, too. And I think it was only there to make the raw reST output look more like some Python docstring markup. But I think we're different enough that keep it isn't that beneficial. So removed. {quote} -- If a package has scripts both w/ and wo/ a further description, they appear separately in the package overview; see, e.g., {{scripts/base/frameworks/packet-filter/index.html}} {quote} Think I fixed it, or at least made it render consistently. {quote} -- should we just omit {{__load__.bro}} in the package overviews? {quote} Not sure, but I think it's easiest to just treat them no different than other scripts. In some cases there might actually be stuff other than {{@load}} in there that would be useful for a person browsing docs (or maybe even having quick access to the set of {{@load}} that is done is useful for quickly switching to other docs). {quote} -- I like how the records now show fields that are redefined somewhere else. Two further thoughts: --- how about changing {{from redef in XXX}} to {{added if XXX is loaded}}. For a user, it might be more intuitive to talk about loading another script than just redef. {quote} Sounds good, changed to {{present if XXX is loaded}}, which seemed clearer that it's referring to the existence of the field. {quote} --- for the script that does the redef, can we show the new fields there as well? For example, in {{scripts/policy/protocols/http/software-browser-plugins.bro.html}} it would show not only that {{HTTP::Info}} got redef, but also repeat the fields it added right there. Same for the enums. {quote} Should be technically possible (all the info is tracked), but there'd be some effort setting up new linkage to the doc. Also not sure how confusing it is to have pieces of the total type put in various places -- right now there's always one canonical place for the full type and there's no duplication of any documentation. Holding off on doing anything for now. Updated the /topic/jsiwek/broxygen branch with the changes. > topic/jsiwek/broxygen > --------------------- > > Key: BIT-1098 > URL: https://bro-tracker.atlassian.net/browse/BIT-1098 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, Broccoli > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > This branch is in the bro and broccoli repos and improves the automated script-reference documentation generation process, broxygen. > This should address issues in BIT-701 and BIT-751, let me know if there's anything not covered that's still relevant. > Highlights: > - Remove {{--doc-scripts}} and {{-Z}} options to toggle documentation mode -- the parser is now always instrumented to gather documentation from comments of the form "##", "##!", or "##<". > - Raw comments are available at runtime through several BIF functions: {{get_*_comments}}; > - Add {{--broxygen}} and {{-X}} options to toggle generating reST-format documentation output, driven by a config file argument. > - Add a "broxygen" Sphinx extension domain, allowing certain pieces of documentation to be generated on-the-fly via invoking a Bro process. Re-organized/cleaned up the Sphinx source tree in {{doc/}} to use this in some places. -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Fri Dec 6 07:58:45 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 6 Dec 2013 09:58:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1098) topic/jsiwek/broxygen In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1098: --------------------------- Status: Merge Request (was: Open) > topic/jsiwek/broxygen > --------------------- > > Key: BIT-1098 > URL: https://bro-tracker.atlassian.net/browse/BIT-1098 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, Broccoli > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > This branch is in the bro and broccoli repos and improves the automated script-reference documentation generation process, broxygen. > This should address issues in BIT-701 and BIT-751, let me know if there's anything not covered that's still relevant. > Highlights: > - Remove {{--doc-scripts}} and {{-Z}} options to toggle documentation mode -- the parser is now always instrumented to gather documentation from comments of the form "##", "##!", or "##<". > - Raw comments are available at runtime through several BIF functions: {{get_*_comments}}; > - Add {{--broxygen}} and {{-X}} options to toggle generating reST-format documentation output, driven by a config file argument. > - Add a "broxygen" Sphinx extension domain, allowing certain pieces of documentation to be generated on-the-fly via invoking a Bro process. Re-organized/cleaned up the Sphinx source tree in {{doc/}} to use this in some places. -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Fri Dec 6 09:11:45 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 6 Dec 2013 11:11:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1107) Documentation of BIFs that take variable number of arguments In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1107: --------------------------- Description: The function prototype for BIFs that take a variable number of arguments appears in an altered form in the online documentation. Here is a comparison of how these functions appear in the source code, versus what they look like in the online documentation: md5_hash%(...%) ===> Type : function (va_args: any) order%(v: any, ...%) ===> Type : function (va_args: any) sort%(v: any, ...%) ===> Type : function (va_args: any) cat_sep%(sep: string, def: string, ...%) ===> Type : function (va_args: any) The functions that have a named argument ("v" in sort, or "sep" in cat_sep) have those arguments described in the online documentation, but we cannot see them in the function prototype (only "va_args" is shown, which isn't actually the name of any function argument). was: {quote} md5_hash%(...%) ===> Type : function (va_args: any) {quote} I'm thinking the only thing here would be to expect the documenter to know how the BIF compiler translates variadic functions and to add documentation for the va_args parameters. Otherwise the only thing I can think that might make it more clear would be to change the Bro parser to do variadic functions in a more familiar way. {quote} order%(v: any, ...%) ===> Type : function (va_args: any) sort%(v: any, ...%) ===> Type : function (va_args: any) {quote} These look like a hack to workaround the BIF compiler's limitation of not being able to parse the full range of Bro types, in this case functions. Ideal solution: fix it so full range of Bro types can be parsed in BIFs. Workaround: just improve documentation about how a function with a param of type "any" can be used like a variadic function and for each hacky use of it, make it clear in the docs what the expected arguments are. {quote} cat_sep%(sep: string, def: string, ...%) ===> Type : function (va_args: any) {quote} I'm not sure what would fix this... maybe better variadic function support in Bro is needed so the BIF compiler will just pass the named arguments through instead of combining it into one "va_args: any" ? > Documentation of BIFs that take variable number of arguments > ------------------------------------------------------------ > > Key: BIT-1107 > URL: https://bro-tracker.atlassian.net/browse/BIT-1107 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Daniel Thayer > > The function prototype for BIFs that take a variable number of > arguments appears in an altered form in the online documentation. > Here is a comparison of how these functions appear in the source code, > versus what they look like in the online documentation: > md5_hash%(...%) ===> Type : function (va_args: any) > order%(v: any, ...%) ===> Type : function (va_args: any) > sort%(v: any, ...%) ===> Type : function (va_args: any) > cat_sep%(sep: string, def: string, ...%) ===> Type : function (va_args: any) > The functions that have a named argument ("v" in sort, or "sep" in cat_sep) > have those arguments described in the online documentation, but we > cannot see them in the function prototype (only "va_args" is shown, > which isn't actually the name of any function argument). -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Fri Dec 6 09:11:45 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 6 Dec 2013 11:11:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1107) Documentation of BIFs that take variable number of arguments In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1107: --------------------------- Description: {quote} md5_hash%(...%) ===> Type : function (va_args: any) {quote} I'm thinking the only thing here would be to expect the documenter to know how the BIF compiler translates variadic functions and to add documentation for the va_args parameters. Otherwise the only thing I can think that might make it more clear would be to change the Bro parser to do variadic functions in a more familiar way. {quote} order%(v: any, ...%) ===> Type : function (va_args: any) sort%(v: any, ...%) ===> Type : function (va_args: any) {quote} These look like a hack to workaround the BIF compiler's limitation of not being able to parse the full range of Bro types, in this case functions. Ideal solution: fix it so full range of Bro types can be parsed in BIFs. Workaround: just improve documentation about how a function with a param of type "any" can be used like a variadic function and for each hacky use of it, make it clear in the docs what the expected arguments are. {quote} cat_sep%(sep: string, def: string, ...%) ===> Type : function (va_args: any) {quote} I'm not sure what would fix this... maybe better variadic function support in Bro is needed so the BIF compiler will just pass the named arguments through instead of combining it into one "va_args: any" ? was: The function prototype for BIFs that take a variable number of arguments appears in an altered form in the online documentation. Here is a comparison of how these functions appear in the source code, versus what they look like in the online documentation: md5_hash%(...%) ===> Type : function (va_args: any) order%(v: any, ...%) ===> Type : function (va_args: any) sort%(v: any, ...%) ===> Type : function (va_args: any) cat_sep%(sep: string, def: string, ...%) ===> Type : function (va_args: any) The functions that have a named argument ("v" in sort, or "sep" in cat_sep) have those arguments described in the online documentation, but we cannot see them in the function prototype (only "va_args" is shown, which isn't actually the name of any function argument). > Documentation of BIFs that take variable number of arguments > ------------------------------------------------------------ > > Key: BIT-1107 > URL: https://bro-tracker.atlassian.net/browse/BIT-1107 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Daniel Thayer > > {quote} > md5_hash%(...%) ===> Type : function (va_args: any) > {quote} > I'm thinking the only thing here would be to expect the documenter to know how the BIF compiler translates variadic functions and to add documentation for the va_args parameters. Otherwise the only thing I can think that might make it more clear would be to change the Bro parser to do variadic functions in a more familiar way. > {quote} > order%(v: any, ...%) ===> Type : function (va_args: any) > sort%(v: any, ...%) ===> Type : function (va_args: any) > {quote} > These look like a hack to workaround the BIF compiler's limitation of not being able to parse the full range of Bro types, in this case functions. > Ideal solution: fix it so full range of Bro types can be parsed in BIFs. > Workaround: just improve documentation about how a function with a param of type "any" can be used like a variadic function and for each hacky use of it, make it clear in the docs what the expected arguments are. > {quote} > cat_sep%(sep: string, def: string, ...%) ===> Type : function (va_args: any) > {quote} > I'm not sure what would fix this... maybe better variadic function support in Bro is needed so the BIF compiler will just pass the named arguments through instead of combining it into one "va_args: any" ? -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Fri Dec 6 09:13:45 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 6 Dec 2013 11:13:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1107) Documentation of BIFs that take variable number of arguments In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14915#comment-14915 ] Jon Siwek commented on BIT-1107: -------------------------------- (sorry, meant to comment, not edit the description) {quote} md5_hash%(...%) ===> Type : function (va_args: any) {quote} I'm thinking the only thing here would be to expect the documenter to know how the BIF compiler translates variadic functions and to add documentation for the va_args parameters. Otherwise the only thing I can think that might make it more clear would be to change the Bro parser to do variadic functions in a more familiar way. {quote} order%(v: any, ...%) ===> Type : function (va_args: any) sort%(v: any, ...%) ===> Type : function (va_args: any) {quote} These look like a hack to workaround the BIF compiler's limitation of not being able to parse the full range of Bro types, in this case functions. Ideal solution: fix it so full range of Bro types can be parsed in BIFs. Workaround: just improve documentation about how a function with a param of type "any" can be used like a variadic function and for each hacky use of it, make it clear in the docs what the expected arguments are. {quote} cat_sep%(sep: string, def: string, ...%) ===> Type : function (va_args: any) {quote} I'm not sure what would fix this... maybe better variadic function support in Bro is needed so the BIF compiler will just pass the named arguments through instead of combining it into one "va_args: any" ? > Documentation of BIFs that take variable number of arguments > ------------------------------------------------------------ > > Key: BIT-1107 > URL: https://bro-tracker.atlassian.net/browse/BIT-1107 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Daniel Thayer > > The function prototype for BIFs that take a variable number of > arguments appears in an altered form in the online documentation. > Here is a comparison of how these functions appear in the source code, > versus what they look like in the online documentation: > md5_hash%(...%) ===> Type : function (va_args: any) > order%(v: any, ...%) ===> Type : function (va_args: any) > sort%(v: any, ...%) ===> Type : function (va_args: any) > cat_sep%(sep: string, def: string, ...%) ===> Type : function (va_args: any) > The functions that have a named argument ("v" in sort, or "sep" in cat_sep) > have those arguments described in the online documentation, but we > cannot see them in the function prototype (only "va_args" is shown, > which isn't actually the name of any function argument). -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Fri Dec 6 09:15:45 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 6 Dec 2013 11:15:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-701) Autodoc final version of redef'ed records (for logging) In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-701: -------------------------- Resolution: Fixed Fix Version/s: (was: 2.2) 2.3 Status: Closed (was: Open) > Autodoc final version of redef'ed records (for logging) > ------------------------------------------------------- > > Key: BIT-701 > URL: https://bro-tracker.atlassian.net/browse/BIT-701 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: gregor > Labels: docs > Fix For: 2.3 > > > (from an earlier e-mail) > Hi, > given a bunch of scripts, that extend record types (I mostly thinking > about the Info records used for logging) if there a way to get a autodoc > generated file that shows the final version of the record? > The idea would be, that this "final version" is exactly what actually > gets logged, so it would be a nice-to-have for users who want to > understand all the columns in a log file. Now, they have to know all the > scripts that are loaded that extend a particular record and look at > their documentation to find out the meaning of columns / fields. This is > fine for scripts in policy, but for the base scripts it would be nice to > have a single place to go too. -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Fri Dec 6 09:15:45 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 6 Dec 2013 11:15:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-751) Broxygen Wishlist In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14917#comment-14917 ] Jon Siwek commented on BIT-751: ------------------------------- Addressed in BIT-1098 > Broxygen Wishlist > ----------------- > > Key: BIT-751 > URL: https://bro-tracker.atlassian.net/browse/BIT-751 > Project: Bro Issue Tracker > Issue Type: Task > Components: Bro > Affects Versions: 2.0 > Reporter: Robin Sommer > Fix For: 2.3 > > > Collecting a number of lower priority items here I noticed: > \\- In a script's summary section, would be nice if the namespace linked > bq. to the corresponding index entry that lists all the scripts > bq. contributing to that namespace. > \\- Thoughts on restructuring the summary section: > * Notices: Should list the new Notices and link them to a part in the Detailed Interface that describes them. Could then become the first entry of the Summary. > * Redefinitions: Currently shows the type being modified and the text associated with the new comment, but not the new value itself. Not totally sure how to change, but perhaps just list the ID being modified here and then also link to Detailed INterface section. > * Redefinitions: That's sometimes hard to understand currently, like in ?scripts/base/frameworks/notice/weird.html: It's not that intuitive that Weird::Log is added to Log::ID; and also not what the comment belongs to. > * Package index: would be nice if it had a brief description of each package, ideally generated automatically somehow (from comments in the +load+.bro perhaps?) > \\- make \-j doesn't work reliably with the doc generation, can give some > bq. odd errors. > \\- With all docs in Sphinx now, it would be helpful not have to rebuild > bq. everything (including Broxygen) each time one runs "make doc". I > bq. remember we discussed using the pickled Sphinx cache before and it > bq. didn't seem worth the trouble. May be worth reconsiderng now to keep > bq. the rebuild times small when doing just doc/* changes. > \\- CSS styling the Broxygen part is a bit tricky because it always > bq. impacts the rest of the Sphinx-generated content. Is there a way to > bq. have a general "broxygen" CSS selector to select only Broxygen > bq. items? -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Fri Dec 6 09:15:45 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 6 Dec 2013 11:15:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-701) Autodoc final version of redef'ed records (for logging) In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14916#comment-14916 ] Jon Siwek commented on BIT-701: ------------------------------- This is addressed by BIT-1098 > Autodoc final version of redef'ed records (for logging) > ------------------------------------------------------- > > Key: BIT-701 > URL: https://bro-tracker.atlassian.net/browse/BIT-701 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: gregor > Labels: docs > Fix For: 2.3 > > > (from an earlier e-mail) > Hi, > given a bunch of scripts, that extend record types (I mostly thinking > about the Info records used for logging) if there a way to get a autodoc > generated file that shows the final version of the record? > The idea would be, that this "final version" is exactly what actually > gets logged, so it would be a nice-to-have for users who want to > understand all the columns in a log file. Now, they have to know all the > scripts that are loaded that extend a particular record and look at > their documentation to find out the meaning of columns / fields. This is > fine for scripts in policy, but for the base scripts it would be nice to > have a single place to go too. -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Fri Dec 6 09:15:45 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 6 Dec 2013 11:15:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-751) Broxygen Wishlist In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-751: -------------------------- Resolution: Fixed Fix Version/s: (was: 2.2) 2.3 Status: Closed (was: Open) > Broxygen Wishlist > ----------------- > > Key: BIT-751 > URL: https://bro-tracker.atlassian.net/browse/BIT-751 > Project: Bro Issue Tracker > Issue Type: Task > Components: Bro > Affects Versions: 2.0 > Reporter: Robin Sommer > Fix For: 2.3 > > > Collecting a number of lower priority items here I noticed: > \\- In a script's summary section, would be nice if the namespace linked > bq. to the corresponding index entry that lists all the scripts > bq. contributing to that namespace. > \\- Thoughts on restructuring the summary section: > * Notices: Should list the new Notices and link them to a part in the Detailed Interface that describes them. Could then become the first entry of the Summary. > * Redefinitions: Currently shows the type being modified and the text associated with the new comment, but not the new value itself. Not totally sure how to change, but perhaps just list the ID being modified here and then also link to Detailed INterface section. > * Redefinitions: That's sometimes hard to understand currently, like in ?scripts/base/frameworks/notice/weird.html: It's not that intuitive that Weird::Log is added to Log::ID; and also not what the comment belongs to. > * Package index: would be nice if it had a brief description of each package, ideally generated automatically somehow (from comments in the +load+.bro perhaps?) > \\- make \-j doesn't work reliably with the doc generation, can give some > bq. odd errors. > \\- With all docs in Sphinx now, it would be helpful not have to rebuild > bq. everything (including Broxygen) each time one runs "make doc". I > bq. remember we discussed using the pickled Sphinx cache before and it > bq. didn't seem worth the trouble. May be worth reconsiderng now to keep > bq. the rebuild times small when doing just doc/* changes. > \\- CSS styling the Broxygen part is a bit tricky because it always > bq. impacts the rest of the Sphinx-generated content. Is there a way to > bq. have a general "broxygen" CSS selector to select only Broxygen > bq. items? -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Fri Dec 6 09:33:45 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 6 Dec 2013 11:33:45 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1107) Documentation of BIFs that take variable number of arguments In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14918#comment-14918 ] Robin Sommer commented on BIT-1107: ----------------------------------- The work-around of turning va_args function arguments into {{(...)}}}, along with a manual textual description of how the parameters are supposed to look like in each case, would sound good to me. Btw, I believe this is how Bro recognizes va_args functions: {noformat} int check_and_promote_exprs(ListExpr*& elements, TypeList* types) { [...] if ( tl->length() == 1 && (*tl)[0]->Tag() == TYPE_ANY ) return 1; [...] } {noformat} Would be nicer to have some more explicit way some time. > Documentation of BIFs that take variable number of arguments > ------------------------------------------------------------ > > Key: BIT-1107 > URL: https://bro-tracker.atlassian.net/browse/BIT-1107 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Daniel Thayer > > The function prototype for BIFs that take a variable number of > arguments appears in an altered form in the online documentation. > Here is a comparison of how these functions appear in the source code, > versus what they look like in the online documentation: > md5_hash%(...%) ===> Type : function (va_args: any) > order%(v: any, ...%) ===> Type : function (va_args: any) > sort%(v: any, ...%) ===> Type : function (va_args: any) > cat_sep%(sep: string, def: string, ...%) ===> Type : function (va_args: any) > The functions that have a named argument ("v" in sort, or "sep" in cat_sep) > have those arguments described in the online documentation, but we > cannot see them in the function prototype (only "va_args" is shown, > which isn't actually the name of any function argument). -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From leres at ee.lbl.gov Fri Dec 6 22:22:36 2013 From: leres at ee.lbl.gov (Craig Leres) Date: Fri, 06 Dec 2013 22:22:36 -0800 Subject: [Bro-Dev] Fwd: [REL - 10amd64-default][security/bro] Failed for bro-2.2 in build In-Reply-To: <20131203160954.GG52843@icir.org> References: <201311300808.rAU88xFv086973@beefy2.isc.freebsd.org> <529A961A.4090809@ee.lbl.gov> <20131203160954.GG52843@icir.org> Message-ID: <52A2BEAC.6070605@ee.lbl.gov> On 12/03/13 08:09, Robin Sommer wrote: > Which clang version is this? I've tried it with a recent version of > the clang 3.4 release branch, and that works fine for me. I dunno, I've asked them them to include the clang version before; I'll ask again. > But based on the error message, I'm attaching a patch; does that help > by any chance? I'll see if I can get someone to test it. Thanks! Craig From noreply at bro.org Sat Dec 7 00:00:16 2013 From: noreply at bro.org (Merge Tracker) Date: Sat, 7 Dec 2013 00:00:16 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201312070800.rB780GZk019451@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------------------- -------------- -------------- ---------- ------------- ---------- -------------------------------------- BIT-1106 [1] Bro Bernhard Amann - 2013-12-05 - Normal Merge topic/bernhard/input-error-fixes BIT-1105 [2] Bro,Broccoli,BroControl Jon Siwek - 2013-12-05 2.3 High /topic/jsiwek/misc-fixes BIT-1104 [3] Bro Michael Stone Seth Hall 2013-12-05 - Normal Add tracking for MSIE 11 BIT-1103 [4] Bro Andrew Hoying Bernhard Amann 2013-12-05 - High Memory leak in Bro Intel framework BIT-1098 [5] Bro,Broccoli Jon Siwek - 2013-12-06 2.3 Normal topic/jsiwek/broxygen [6] [1] BIT-1106 https://bro-tracker.atlassian.net/browse/BIT-1106 [2] BIT-1105 https://bro-tracker.atlassian.net/browse/BIT-1105 [3] BIT-1104 https://bro-tracker.atlassian.net/browse/BIT-1104 [4] BIT-1103 https://bro-tracker.atlassian.net/browse/BIT-1103 [5] BIT-1098 https://bro-tracker.atlassian.net/browse/BIT-1098 [6] broxygen https://github.com/bro/bro,broccoli/tree/topic/jsiwek/broxygen From noreply at bro.org Sun Dec 8 00:00:15 2013 From: noreply at bro.org (Merge Tracker) Date: Sun, 8 Dec 2013 00:00:15 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201312080800.rB880FUh027083@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------------------- -------------- -------------- ---------- ------------- ---------- -------------------------------------- BIT-1106 [1] Bro Bernhard Amann - 2013-12-05 - Normal Merge topic/bernhard/input-error-fixes BIT-1105 [2] Bro,Broccoli,BroControl Jon Siwek - 2013-12-05 2.3 High /topic/jsiwek/misc-fixes BIT-1104 [3] Bro Michael Stone Seth Hall 2013-12-05 - Normal Add tracking for MSIE 11 BIT-1103 [4] Bro Andrew Hoying Bernhard Amann 2013-12-05 - High Memory leak in Bro Intel framework BIT-1098 [5] Bro,Broccoli Jon Siwek - 2013-12-06 2.3 Normal topic/jsiwek/broxygen [6] [1] BIT-1106 https://bro-tracker.atlassian.net/browse/BIT-1106 [2] BIT-1105 https://bro-tracker.atlassian.net/browse/BIT-1105 [3] BIT-1104 https://bro-tracker.atlassian.net/browse/BIT-1104 [4] BIT-1103 https://bro-tracker.atlassian.net/browse/BIT-1103 [5] BIT-1098 https://bro-tracker.atlassian.net/browse/BIT-1098 [6] broxygen https://github.com/bro/bro,broccoli/tree/topic/jsiwek/broxygen From noreply at bro.org Mon Dec 9 00:00:16 2013 From: noreply at bro.org (Merge Tracker) Date: Mon, 9 Dec 2013 00:00:16 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201312090800.rB980Gic003220@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------------------- -------------- -------------- ---------- ------------- ---------- -------------------------------------- BIT-1106 [1] Bro Bernhard Amann - 2013-12-05 - Normal Merge topic/bernhard/input-error-fixes BIT-1105 [2] Bro,Broccoli,BroControl Jon Siwek - 2013-12-05 2.3 High /topic/jsiwek/misc-fixes BIT-1104 [3] Bro Michael Stone Seth Hall 2013-12-05 - Normal Add tracking for MSIE 11 BIT-1103 [4] Bro Andrew Hoying Bernhard Amann 2013-12-05 - High Memory leak in Bro Intel framework BIT-1098 [5] Bro,Broccoli Jon Siwek - 2013-12-06 2.3 Normal topic/jsiwek/broxygen [6] [1] BIT-1106 https://bro-tracker.atlassian.net/browse/BIT-1106 [2] BIT-1105 https://bro-tracker.atlassian.net/browse/BIT-1105 [3] BIT-1104 https://bro-tracker.atlassian.net/browse/BIT-1104 [4] BIT-1103 https://bro-tracker.atlassian.net/browse/BIT-1103 [5] BIT-1098 https://bro-tracker.atlassian.net/browse/BIT-1098 [6] broxygen https://github.com/bro/bro,broccoli/tree/topic/jsiwek/broxygen From jira at bro-tracker.atlassian.net Mon Dec 9 15:47:52 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 9 Dec 2013 17:47:52 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1105) /topic/jsiwek/misc-fixes In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1105: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > /topic/jsiwek/misc-fixes > ------------------------ > > Key: BIT-1105 > URL: https://bro-tracker.atlassian.net/browse/BIT-1105 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro, Broccoli, BroControl > Affects Versions: git/master > Reporter: Jon Siwek > Priority: High > Fix For: 2.3 > > > This is in bro, broccoli, and broctl. It fixes various build/test/coverity failures. > The ref counting fix may be a pre-existing issue relevant to 2.2, but just coincidentally exposed on one jenkins node now. -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Mon Dec 9 15:47:52 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 9 Dec 2013 17:47:52 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1098) topic/jsiwek/broxygen In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1098: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > topic/jsiwek/broxygen > --------------------- > > Key: BIT-1098 > URL: https://bro-tracker.atlassian.net/browse/BIT-1098 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, Broccoli > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > This branch is in the bro and broccoli repos and improves the automated script-reference documentation generation process, broxygen. > This should address issues in BIT-701 and BIT-751, let me know if there's anything not covered that's still relevant. > Highlights: > - Remove {{--doc-scripts}} and {{-Z}} options to toggle documentation mode -- the parser is now always instrumented to gather documentation from comments of the form "##", "##!", or "##<". > - Raw comments are available at runtime through several BIF functions: {{get_*_comments}}; > - Add {{--broxygen}} and {{-X}} options to toggle generating reST-format documentation output, driven by a config file argument. > - Add a "broxygen" Sphinx extension domain, allowing certain pieces of documentation to be generated on-the-fly via invoking a Bro process. Re-organized/cleaned up the Sphinx source tree in {{doc/}} to use this in some places. -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Mon Dec 9 15:47:52 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 9 Dec 2013 17:47:52 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1104) Add tracking for MSIE 11 In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1104: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Add tracking for MSIE 11 > ------------------------ > > Key: BIT-1104 > URL: https://bro-tracker.atlassian.net/browse/BIT-1104 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.1 > Environment: Ubuntu > Reporter: Michael Stone > Assignee: Seth Hall > Labels: analyzer > > MSIE 11.0 currently shows up as . It looks like MS might have changed it's user agent string and doesn't include "MSIE". I added the following to /usr/local/bro/share/bro/base/frameworks/software/main.bro > just below the "MSIE" block and above the "Safari" block. > else if ( /Trident\/7.0/ in uparsed_version ) > { > if ( /rv:11\.0/ in unparsed_version ) { > software_name = "MSIE"; > v = [$major=11,$minor=0]; > } > } > Disclaimer: I'm fairly new to working with Bro so this might not be the best way, but it seems to be working for me. > Thanks! -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Mon Dec 9 15:47:52 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 9 Dec 2013 17:47:52 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1106) Merge topic/bernhard/input-error-fixes In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1106: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Merge topic/bernhard/input-error-fixes > -------------------------------------- > > Key: BIT-1106 > URL: https://bro-tracker.atlassian.net/browse/BIT-1106 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > > The branch topic/bernhard/input-error-fixes fixes a number of issues of the input framework that all have to do with errors: > -First: > Due to architectural constraints, it is very hard for the input framework to handle optional records. For an optional record, either the whole record has to be missing, or all non-optional elements of the record have to be defined. This information is not available to input readers after the records have been unrolled into the threading types. > Behavior so far was to treat optional records like they are non-optional, without warning. The patch changes this behavior to emit an error on stream-creation (during type-checking) and refusing to open the file. I think this is a better idea - the behavior so far was undocumented and unintuitive. > - Second: > For table and event streams, reader backend creation was done very early, before actually checking if all arguments are valid. Initialization is moved after the checks now - this makes a number of delete statements unnecessary. Also - I suspect threads of failed input reader instances were not deleted until shutdown > - Third: > Add a couple more consistency checks, e.g. checking if the destination value of a table has the same type as we need. We did not check everything in all instances, instead we just assigned the things without caring (which works, but is not really desirable). > This change also exposed a few bugs in other testcases where table definitions were wrong (did not respect $want_record) > - Fourth: > Improve error messages and write testcases for all error messages (I think). -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Mon Dec 9 15:47:52 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 9 Dec 2013 17:47:52 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1103) Memory leak in Bro Intel framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1103: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Memory leak in Bro Intel framework > ---------------------------------- > > Key: BIT-1103 > URL: https://bro-tracker.atlassian.net/browse/BIT-1103 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Environment: Red Hat Enterprise Linux Server release 6.5 > Reporter: Andrew Hoying > Assignee: Bernhard Amann > Priority: High > Labels: intel, leak > > The policy/frameworks/intel/seen bro scripts have a memory leak. On my moderately busy Bro installation I am leaking about a gig of memory a day per worker process with the Intel framework enabled. I can replicate by adding the following to the local.bro default script and then running through a small PCAP with primarily dns, dhcp and syslog traffic. > {{ > @load policy/frameworks/intel/seen > redef Intel::read_files += { > "/usr/local/bro/spool/domain_suspicious.txt", > }; > }} > The intel file is in the following format, here's a few sample lines. It is generated automatically by CIF: > {{ > #fields indicator indicator_type meta.source meta.desc meta.url meta.cif_impact meta.cif_severity meta.cif_confidence > mete-tools.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=mete-tools.biz (public) - - 95 > rttvxygkmwlqmq.net Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=rttvxygkmwlqmq.net (public) - - 95 > podserveruho.com Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=podserveruho.com (public) - - 95 > wwfcogdgntlxw.biz Intel::DOMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=wwfcogdgntlxw.biz (public) - - 95 > }} > I compiled bro with gperftool debug support and followed the instructions here: http://www.bro.org/development/howtos/leaks.html. (Note, the instructions are wrong on the flags for ./configure, you need to add --enable-perftools-debug to get the -m option for bro) > Here's the output from pprof top after running a PCAP trace with 10,000 packets. Running traces with more packets show a greater number of lost objects in the same code locations. > {{ > # pprof bin/bro "/tmp/bro.24541.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 > Using local file bin/bro. > Using local file /tmp/bro.24541.net_run-end.heap. > Welcome to pprof! For help, type 'help'. > (pprof) top > Total: 4295 objects > 2150 50.1% 50.1% 2150 50.1% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:186 > 2141 49.8% 99.9% 2141 49.8% copy_string /usr/src/bro-2.2/src/util.cc:155 > 2 0.0% 100.0% 2 0.0% re_alloc /usr/src/bro-2.2/build/src/re-scan.cc:2287 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:110 > 1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:133 > 0 0.0% 100.0% 2141 49.8% AsciiFormatter::ParseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:195 > 0 0.0% 100.0% 4 0.1% Connection::NextPacket /usr/src/bro-2.2/src/Conn.cc:259 > 0 0.0% 100.0% 4 0.1% NetSessions::DispatchPacket /usr/src/bro-2.2/src/Sessions.cc:189 > 0 0.0% 100.0% 4 0.1% NetSessions::DoNextPacket /usr/src/bro-2.2/src/Sessions.cc:709 > 0 0.0% 100.0% 4 0.1% NetSessions::NextPacket /usr/src/bro-2.2/src/Sessions.cc:247 > }} -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Tue Dec 10 12:36:50 2013 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Tue, 10 Dec 2013 14:36:50 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1108) Add broctl option to set PF_RING cluster type In-Reply-To: References: Message-ID: Daniel Thayer created BIT-1108: ---------------------------------- Summary: Add broctl option to set PF_RING cluster type Key: BIT-1108 URL: https://bro-tracker.atlassian.net/browse/BIT-1108 Project: Bro Issue Tracker Issue Type: Problem Components: BroControl Reporter: Daniel Thayer Currently, when using PF_RING, broctl chooses the PF_RING cluster type by setting the environment variable PCAP_PF_RING_USE_CLUSTER_PER_FLOW. In order to use a different cluster type, we would need to set a different environment variable (the PF_RING-aware libpcap does not look at the actual value of the environment variable, just whether the variable is defined or not), but there is no option in broctl to do this. To address this issue, a new broctl option PFRINGClusterType can be added, then a user could change the value of this option to choose a different PF_RING cluster type (and the broctl pf_ring plugin would set the appropriate env. variable). The allowed values of this new broctl option would be: "2-tuple", "4-tuple", "5-tuple", "tcp-5-tuple", "round-robin", or "6-tuple" (this one corresponds to the current cluster type used by broctl). By default, PFRINGClusterType would be set to "6-tuple". -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From robin at icir.org Wed Dec 11 10:48:46 2013 From: robin at icir.org (Robin Sommer) Date: Wed, 11 Dec 2013 10:48:46 -0800 Subject: [Bro-Dev] Proposed IOSource reorg In-Reply-To: <663C4ECE-3500-45DC-94B7-8962ABA8391E@icir.org> References: <20131203180746.GD69466@icir.org> <663C4ECE-3500-45DC-94B7-8962ABA8391E@icir.org> Message-ID: <20131211184846.GA30343@icir.org> As I'm working on the reorg, I propose to do the following: - Remove flow sources completely for now. Per below, we should eventually turn them into a file analyzer and at it doesn't look worth the effort (nor the ugliness) to migrate them over to the new structure first only to throw them out later. I'd be surprised if anybody is using them anyways. - Remove the secondary path from the packet-layer code. We have discussed this before and at that time decided for keeping the code; see https://bro-tracker.atlassian.net/browse/BIT-434 However, I propose to go ahead and remove now because (1) it doesn't really fit the new structure of making the API (mostly) pcap-independent (it never really fit in well in the first place, and has made the code a lot more complex); (2) large-conns.bro seems to be the only actual use case, which we don't ship with 2.x anymore, and I'm not convinced that by itself warrants a separate data path (can we find a different solution to the problem?); and (3) it would be quite a bit of additional effort to port the code and make sure it still works (we don't have any tests, not surprisingly). Thoughts? Robin On Wed, Dec 04, 2013 at 11:12 -0500, you wrote: > > On Dec 3, 2013, at 1:07 PM, Robin Sommer wrote: > > > src/iosource/sources/flow-src/* > > To document our conversation from yesterday, flow-src should probably > be thrown out and the netflow analyzer turned into a file analyzer. > Extending the input framework to be able to open raw sockets would > then enable us to create an input stream holding open a datagram > socket and attach the netflow file analyzer to it. This would > simplify the whole thing and make it possible to reuse the netflow > analyzer code because we could yank netflow directly off the wire with > it too (pending some analyzer infrastructure re-architecting). > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From seth at icir.org Wed Dec 11 11:23:12 2013 From: seth at icir.org (Seth Hall) Date: Wed, 11 Dec 2013 14:23:12 -0500 Subject: [Bro-Dev] Proposed IOSource reorg In-Reply-To: <20131211184846.GA30343@icir.org> References: <20131203180746.GD69466@icir.org> <663C4ECE-3500-45DC-94B7-8962ABA8391E@icir.org> <20131211184846.GA30343@icir.org> Message-ID: <2A40D84A-901E-47D7-AE4A-05CBCDAFB208@icir.org> On Dec 11, 2013, at 1:48 PM, Robin Sommer wrote: > As I'm working on the reorg, I propose to do the following: Everything sound good to me. > large-conns.bro seems to be the only actual use case, which we > don't ship with 2.x anymore, and I'm not convinced that by > itself warrants a separate data path (can we find a different > solution to the problem?) I think the only reason that large-conns.bro used the secondary data path was to get access to more traffic that Bro might not be normally seeing due to being filtered out. With our change to a default open filter (and I don't see this changing anytime soon) we don't need the secondary path to let in more traffic than is normally allowed in. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20131211/e12d140a/attachment.bin From jira at bro-tracker.atlassian.net Wed Dec 11 18:20:51 2013 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Wed, 11 Dec 2013 20:20:51 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1108) Add broctl option to set PF_RING cluster type In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000#comment-15000 ] Daniel Thayer commented on BIT-1108: ------------------------------------ Any opinions on whether we should keep the same default cluster type (6-tuple) as before, or change it to something else? > Add broctl option to set PF_RING cluster type > --------------------------------------------- > > Key: BIT-1108 > URL: https://bro-tracker.atlassian.net/browse/BIT-1108 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Reporter: Daniel Thayer > > Currently, when using PF_RING, broctl chooses the PF_RING > cluster type by setting the environment variable > PCAP_PF_RING_USE_CLUSTER_PER_FLOW. In order to use a > different cluster type, we would need to set a different > environment variable (the PF_RING-aware libpcap does not > look at the actual value of the environment variable, > just whether the variable is defined or not), but there is > no option in broctl to do this. > To address this issue, a new broctl option PFRINGClusterType > can be added, then a user could change the value of this > option to choose a different PF_RING cluster type (and the > broctl pf_ring plugin would set the appropriate env. variable). > The allowed values of this new broctl option would be: > "2-tuple", "4-tuple", "5-tuple", "tcp-5-tuple", "round-robin", > or "6-tuple" (this one corresponds to the current > cluster type used by broctl). By default, PFRINGClusterType > would be set to "6-tuple". -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From jira at bro-tracker.atlassian.net Wed Dec 11 18:27:51 2013 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Wed, 11 Dec 2013 20:27:51 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1108) Add broctl option to set PF_RING cluster type In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001#comment-15001 ] Seth Hall commented on BIT-1108: -------------------------------- Let's change the default to 4-tuple. > Add broctl option to set PF_RING cluster type > --------------------------------------------- > > Key: BIT-1108 > URL: https://bro-tracker.atlassian.net/browse/BIT-1108 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Reporter: Daniel Thayer > > Currently, when using PF_RING, broctl chooses the PF_RING > cluster type by setting the environment variable > PCAP_PF_RING_USE_CLUSTER_PER_FLOW. In order to use a > different cluster type, we would need to set a different > environment variable (the PF_RING-aware libpcap does not > look at the actual value of the environment variable, > just whether the variable is defined or not), but there is > no option in broctl to do this. > To address this issue, a new broctl option PFRINGClusterType > can be added, then a user could change the value of this > option to choose a different PF_RING cluster type (and the > broctl pf_ring plugin would set the appropriate env. variable). > The allowed values of this new broctl option would be: > "2-tuple", "4-tuple", "5-tuple", "tcp-5-tuple", "round-robin", > or "6-tuple" (this one corresponds to the current > cluster type used by broctl). By default, PFRINGClusterType > would be set to "6-tuple". -- This message was sent by Atlassian JIRA (v6.2-OD-03#6206) From noreply at bro.org Thu Dec 12 00:00:15 2013 From: noreply at bro.org (Merge Tracker) Date: Thu, 12 Dec 2013 00:00:15 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201312120800.rBC80FV4011014@bro-ids.icir.org> Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- --------- ---------- ----------------------------------------------------------- 63c36d5 [1] bro Jon Siwek 2013-12-11 Another attempt to improve core.when-interpreter-exceptions [1] 63c36d5 https://github.com/bro/bro/commit/63c36d58f3c622238a84d0d32b854d33ed5ccc9b From noreply at bro.org Fri Dec 13 00:00:13 2013 From: noreply at bro.org (Merge Tracker) Date: Fri, 13 Dec 2013 00:00:13 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201312130800.rBD80Dpd029601@bro-ids.icir.org> Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- --------- ---------- ------------------------------------------------------- 8ea56ae [1] bro Jon Siwek 2013-12-12 Improve warnings emitted from raw/execute input reader. [1] 8ea56ae https://github.com/bro/bro/commit/8ea56ae567c3bd7c1d7f6349c0507ef6afef3d4d From noreply at bro.org Sat Dec 14 00:00:24 2013 From: noreply at bro.org (Merge Tracker) Date: Sat, 14 Dec 2013 00:00:24 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201312140800.rBE80Olu003400@bro-ids.icir.org> Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- --------- ---------- ------------------------------------------------------- 8ea56ae [1] bro Jon Siwek 2013-12-12 Improve warnings emitted from raw/execute input reader. [1] 8ea56ae https://github.com/bro/bro/commit/8ea56ae567c3bd7c1d7f6349c0507ef6afef3d4d From noreply at bro.org Sun Dec 15 00:00:11 2013 From: noreply at bro.org (Merge Tracker) Date: Sun, 15 Dec 2013 00:00:11 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201312150800.rBF80BJU009661@bro-ids.icir.org> Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- --------- ---------- ------------------------------------------------------- 8ea56ae [1] bro Jon Siwek 2013-12-12 Improve warnings emitted from raw/execute input reader. [1] 8ea56ae https://github.com/bro/bro/commit/8ea56ae567c3bd7c1d7f6349c0507ef6afef3d4d From noreply at bro.org Mon Dec 16 00:00:13 2013 From: noreply at bro.org (Merge Tracker) Date: Mon, 16 Dec 2013 00:00:13 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201312160800.rBG80D5L014176@bro-ids.icir.org> Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- --------- ---------- ------------------------------------------------------- 8ea56ae [1] bro Jon Siwek 2013-12-12 Improve warnings emitted from raw/execute input reader. [1] 8ea56ae https://github.com/bro/bro/commit/8ea56ae567c3bd7c1d7f6349c0507ef6afef3d4d From robin at icir.org Mon Dec 16 12:34:32 2013 From: robin at icir.org (Robin Sommer) Date: Mon, 16 Dec 2013 12:34:32 -0800 Subject: [Bro-Dev] Dynamic plugin model (Re: [Bro-Commits] [git/bro] topic/robin/dynamic-plugins-2.3: Start of a plugin writing how-to. (87a1618)) In-Reply-To: <201312162009.rBGK9eUu019708@bro-ids.icir.org> References: <201312162009.rBGK9eUu019708@bro-ids.icir.org> Message-ID: <20131216203432.GA63319@icir.org> I'd appreciate feedback on the model for dynamic plugins described below. The document contains a short intro on writing a simple plugin, as well as some background on how/when Bro integrates plugins. The code is in topic/robin/dynamic-plugins-2.3. It hasn't seen much testing yet but the basic infrastructure seems to work on Linux and Darwin at least. Robin On Mon, Dec 16, 2013 at 12:09 -0800, I wrote: > +=================== > +Writing Bro Plugins > +=================== > + > +Bro is internally moving to a plugin structure that enables extending > +the system dynamically, without modifying the core code base. That way > +custom code remains self-contained and can be maintained, compiled, > +and installed independently. Currently, plugins can add the following > +functionality to Bro: > + > + - Bro scripts. > + > + - Builtin functions/events/types for the scripting language. > + > + - Protocol and file analyzers. > + > + - Packet sources and packet dumpers. TODO: Not yet. > + > + - Logging framework backends. TODO: Not yet. > + > + - Input framework readers. TODO: Not yet. > + > +A plugin's functionality is available to the user just as if Bro had > +the corresponding code built-in. Indeed, internally many of Bro's > +pieces are structured as plugins as well, they are just statically > +compiled into the binary rather than loaded dynamically at runtime, as > +external plugins are. > + > +Quick Start > +=========== > + > +Writing a basic plugin is quite straight-forward as long as one > +follows a few conventions. In the following we walk through adding a > +new built-in function (bif) to Bro; we'll add `a `rot13(s: string) : > +string``, a function that rotates every character in a string by 13 > +places. > + > +A plugin comes in the form of a directory following a certain > +structure. To get started, Bro's distribution provides a helper script > +``aux/bro-aux/plugin-support/init-plugin`` that creates a skeleton > +plugin that can then be customized. Let's use that:: > + > + # mkdir rot13-plugin > + # cd rot13-plugin > + # init-plugin Demo Rot13 > + > +As you can see the script takes two arguments. The first is a > +namespace the plugin will live in, and the second a descriptive name > +for the plugin itself. Bro uses the combination of the two to identify > +a plugin. The namespace serves to avoid naming conflicts between > +plugins written by independent developers; pick, e.g., the name of > +your organisation (and note that the namespace ``Bro`` is reserved for > +functionality distributed by the Bro Project). In our example, the > +plugin will be called ``Demo::Rot13``. > + > +The ``init-plugin`` script puts a number of files in place. The full > +layout is described later. For now, all we need is > +``src/functions.bif``. It's initially empty, but we'll add our new bif > +there as follows:: > + > + # cat scripts/functions.bif > + module CaesarCipher; > + > + function camel_case%(s: string%) : string > + %{ > + char* rot13 = copy_string(s->CheckString()); > + > + for ( char* p = rot13; *p; p++ ) > + { > + char b = islower(*p) ? 'a' : 'A'; > + *p = (*p - b + 13) % 26 + b; > + } > + > + return new StringVal(strlen(rot13), rot13); > + %} > + > +The syntax of this file is just like any other ``*.bif`` file; we > +won't go into it here. > + > +Now we can already compile our plugin, we just need to tell the > +Makefile put in place by ``init-plugin`` where the Bro source tree is > +located (Bro needs to have been built there first):: > + > + # make BRO=/path/to/bro/dist > + [... cmake output ...] > + > +Now our ``rot13-plugin`` directory has everything that it needs > +for Bro to recognize it as a dynamic plugin. Once we point Bro to it, > +it will pull it in automatically, as we can check with the ``-N`` > +option: > + > + # export BRO_PLUGIN_PATH=/path/to/rot13-plugin > + # bro -N > + [...] > + Plugin: Demo::Rot13 - (dynamic, version 1) > + [...] > + > +That looks quite good, except for the dummy description that we should > +replace with something nicer so that users will know what our plugin > +is about. We do this by editing the ``BRO_PLUGIN_DESCRIPTION`` line > +in ``src/Plugin.cc``, like this: > + > + # cat src/Plugin.cc > + > + #include > + > + BRO_PLUGIN_BEGIN(Demo, Rot13) > + BRO_PLUGIN_VERSION(1); > + BRO_PLUGIN_DESCRIPTION("Caesar cipher rotating a string's characters by 13 places."); > + BRO_PLUGIN_BIF_FILE(consts); > + BRO_PLUGIN_BIF_FILE(events); > + BRO_PLUGIN_BIF_FILE(functions); > + BRO_PLUGIN_END > + > + # make > + [...] > + # bro -N | grep Rot13 > + Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1) > + > +Better. Bro can also show us what exactly the plugin provides with the > +more verbose option ``-NN``:: > + > + # bro -NN > + [...] > + Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1) > + [Function] CaesarCipher::rot13 > + [...] > + > +There's our function. Now let's use it:: > + > + # bro -e 'print CaesarCipher::rot13("Hello")' > + Uryyb > + > +It works. We next install the plugin along with Bro itself, so that it > +will find it directly without needing the ``BRO_PLUGIN_PATH`` > +environment variable. If we first unset the variable, the function > +will no longer be available:: > + > + # unset BRO_PLUGIN_PATH > + # bro -e 'print CaesarCipher::rot13("Hello")' > + error in , line 1: unknown identifier CaesarCipher::rot13, at or near "CaesarCipher::rot13" > + > +Once we install it, it works again:: > + > + # make install > + # bro -e 'print CaesarCipher::rot13("Hello")' > + Uryyb > + > +The installed version went into > +``/lib/bro/plugins/Demo_Rot13``. > + > +We can distribute the plugin in either source or binary form by using > +the Makefile's ``sdist`` and ``bdist`` target, respectively. Both > +create corrsponding tarballs:: > + > + # make sdist > + [...] > + Source distribution in build/sdist/Demo_Rot13.tar.gz > + > + # make bdist > + [...] > + Binary distribution in build/Demo_Rot13-darwin-x86_64.tar.gz > + > +The source archive will contain everything in the plugin directory > +except any generated files. The binary archive will contain anything > +needed to install and run the plugin, i.e., just what ``make install`` > +puts into place as well. As the binary distribution is > +platform-dependent, its name includes the OS and architecture the > +plugin was built on. > + > +Plugin Directory Layout > +======================= > + > +A plugin's directory needs to follow a set of conventions so that Bro > +(1) recognizes it as a plugin, and (2) knows what to load. While > +``init-plugin`` takes care of most of this, the following is the full > +story. We'll use ```` to represent a plugin's top-level > +directory. > + > +``/__bro_plugin__`` > + A file that marks a directory as containing a Bro plugin. The file > + must exist, and its content must consist of a single line with the > + qualified name of the plugin (e.g., "Demo::Rot13"). > + > +``/lib/--.so`` > + The shared library containing the plugin's compiled code. Bro will > + load this in dynamically at run-time if OS and architecture match > + the current platform. > + > +``lib/bif/`` > + Directory with auto-generated Bro scripts that declare the plugins > + bif elements. The files here are produced by ``bifcl``. > + > +``scripts/`` > + A directory with the plugin's custom Bro scripts. When the plugin > + gets activated, this directory will be automatically added to > + ``BROPATH``, so that any scripts/modules inside can be > + ``@load``ed. > + > +``scripts``/__load__.bro > + A Bro script that will be loaded immediately when the plugin gets > + activated. See below for more information on activating plugins. > + > +By convention, a plugin should put its custom scripts into sub folders > +of ``scripts/``, i.e., ``scripts//