From bro at tracker.bro-ids.org Tue Nov 1 07:35:40 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 01 Nov 2011 14:35:40 -0000 Subject: [Bro-Dev] #655: broctl 'scripts' command doesn't work in cluster mode Message-ID: <048.94750b9656841ef5d3bd2012524def39@tracker.bro-ids.org> #655: broctl 'scripts' command doesn't work in cluster mode ------------------------+------------------------ Reporter: jsiwek | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: git/master Keywords: | ------------------------+------------------------ `broctl scripts` output: {{{ [...] proxy-1 is ok. cat: loaded_scripts*: No such file or directory worker-1 is ok. cat: loaded_scripts*: No such file or directory worker-2 is ok. cat: loaded_scripts*: No such file or directory }}} Which I think is because those cluster node types disable local logging. Is there a way to set `Log::enable_local_logging=F`, but whitelist the `loaded_scripts.log` stream? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Nov 1 07:41:27 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 01 Nov 2011 14:41:27 -0000 Subject: [Bro-Dev] #655: broctl 'scripts' command doesn't work in cluster mode In-Reply-To: <048.94750b9656841ef5d3bd2012524def39@tracker.bro-ids.org> References: <048.94750b9656841ef5d3bd2012524def39@tracker.bro-ids.org> Message-ID: <063.929aa26a6d8be5a46c38d3c170f1197c@tracker.bro-ids.org> #655: broctl 'scripts' command doesn't work in cluster mode -------------------------+------------------------ Reporter: jsiwek | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: git/master Resolution: | Keywords: -------------------------+------------------------ Comment (by robin): On Tue, Nov 01, 2011 at 14:35 -0000, you wrote: > Is there a way to set `Log::enable_local_logging=F`, but whitelist the > `loaded_scripts.log` stream? Yes, log filters have a "log_local" attribute. Its default is enable_local_logging but that can be overidden. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Nov 1 07:47:31 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 01 Nov 2011 14:47:31 -0000 Subject: [Bro-Dev] #656: Add stats.bro back Message-ID: <047.2d9bdf13c6cee04531b4645e09da8cb7@tracker.bro-ids.org> #656: Add stats.bro back ------------------------+-------------------- Reporter: robin | Type: Task Status: new | Priority: Normal Milestone: Bro2.0 | Component: Bro Version: git/master | ------------------------+-------------------- That didn't make it into the new set of scripts so far, but should be pretty straight-forward to put back in place. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Tue Nov 1 08:17:31 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 1 Nov 2011 08:17:31 -0700 Subject: [Bro-Dev] Notice::Info$src Message-ID: <20111101151731.GI17511@icir.org> I know we've talked about this a few times, but I don't remember what the final verdict was: what's the semantics of the "src" field in Notice::Info now? The comment says: ## Source address, if we don't have a :bro:type:`conn_id`. src: addr &log &optional; But the Server_Found notice sets it like this: NOTICE([$note=Server_Found, $msg=fmt("%s: %s server on port %s%s", c$id$resp_h, s, c$id$resp_p, (known ? " (update)" : "")), $p=c$id$resp_p, $sub=s, $conn=c, $src=c$id$resp_h, $n=a]); Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Tue Nov 1 08:46:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 01 Nov 2011 15:46:07 -0000 Subject: [Bro-Dev] #655: broctl 'scripts' command doesn't work in cluster mode In-Reply-To: <048.94750b9656841ef5d3bd2012524def39@tracker.bro-ids.org> References: <048.94750b9656841ef5d3bd2012524def39@tracker.bro-ids.org> Message-ID: <063.d4aef5d46f7355f6a04f526e4e550f45@tracker.bro-ids.org> #655: broctl 'scripts' command doesn't work in cluster mode -------------------------+------------------------ Reporter: jsiwek | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: git/master Resolution: | Keywords: -------------------------+------------------------ Comment (by seth): > Yes, log filters have a "log_local" attribute. Its default is > enable_local_logging but that can be overidden. Alternately, could we use a helper and just run Bro and get the output of that script instead of just cat-ing a log file? More like how it was done previously with the command line argument. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Tue Nov 1 08:57:27 2011 From: seth at icir.org (Seth Hall) Date: Tue, 1 Nov 2011 11:57:27 -0400 Subject: [Bro-Dev] Notice::Info$src In-Reply-To: <20111101151731.GI17511@icir.org> References: <20111101151731.GI17511@icir.org> Message-ID: <635DF2B4-AFCD-41BE-AE71-9DDBE07CE0DC@icir.org> On Nov 1, 2011, at 11:17 AM, Robin Sommer wrote: > I know we've talked about this a few times, but I don't remember what > the final verdict was: what's the semantics of the "src" field in > Notice::Info now? More or less I stuck with the semantics that you had laid down. $src is dually used (and obviously not documented well inline as you noticed) as the "source" address for packets/connection, but in the event that the notice is an attribute of a host the $src field is used for holding that address. I don't feel like it's very optimal, but so far it's actually been working pretty well. Are you reading the notice framework documentation I pushed out last night? > But the Server_Found notice sets it like this: > > NOTICE([$note=Server_Found, > $msg=fmt("%s: %s server on port %s%s", c$id$resp_h, s, c$id$resp_p, (known ? " (update)" : "")), > $p=c$id$resp_p, $sub=s, $conn=c, > $src=c$id$resp_h, $n=a]); This is due to lingering inconsistency with notices that were written before I really started getting little conceptual issues like this surrounding the notice framework nailed down. We should probably make another round of assessment of the notices before 2.0-release. This was partly why I really wanted to get the new notice framework documented, I wanted everyone else to understand the slightly refined approach to notices I was taking. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Tue Nov 1 09:34:18 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 1 Nov 2011 09:34:18 -0700 Subject: [Bro-Dev] Notice::Info$src In-Reply-To: <635DF2B4-AFCD-41BE-AE71-9DDBE07CE0DC@icir.org> References: <20111101151731.GI17511@icir.org> <635DF2B4-AFCD-41BE-AE71-9DDBE07CE0DC@icir.org> Message-ID: <20111101163418.GL17511@icir.org> On Tue, Nov 01, 2011 at 11:57 -0400, you wrote: > Are you reading the notice framework documentation I pushed out last night? Will do. This came up when I saw alarm summaries claiming Server_Founds to be about a connection "A -> A" due to using $src for the origniator. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Tue Nov 1 10:00:53 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 01 Nov 2011 17:00:53 -0000 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> Message-ID: <062.58752c9fc30263724768d2cc2414705e@tracker.bro-ids.org> #388: Fix more compiler warnings ---------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Task | Status: assigned Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ---------------------+---------------------- Changes (by jsiwek): * owner: => jsiwek * status: new => assigned -- Ticket URL: Bro Tracker Bro Issue Tracker From mcholste at gmail.com Tue Nov 1 18:51:09 2011 From: mcholste at gmail.com (Martin Holste) Date: Tue, 1 Nov 2011 20:51:09 -0500 Subject: [Bro-Dev] snaplen and drops In-Reply-To: <6655064A-DED1-47A5-B7AB-F12868211577@net.in.tum.de> References: <20111027042458.GA87273@icir.org> <535469B6-200D-4940-A2E5-704C89457850@net.in.tum.de> <20111028033646.GA90146@icir.org> <20111028155635.GB12238@icir.org> <6655064A-DED1-47A5-B7AB-F12868211577@net.in.tum.de> Message-ID: > I'm a bit puzzled. If I understand things correctly, libpcap-1.0.0 uses AF_PACKET by default (after checking that MMAP support is available in the running kernel). > I don't think that's how it works, but I'm not a kernel-hacking guru. The reason I'm pretty sure it doesn't work that way is that both Snort and Suricata IDS include separate data acquisition code for libpcap and af_packet, which is nonsensical if you can get af_packet via libpcap natively. I did a bit of Googling and cannot find anything definitive one way or the other. > Cool, I wasn't aware of load balancing features in the standard kernel. Did you do some experiments to compare the standard kernel load-balancing to the one provided by PF_RING? > None of the major distros are using the 3.0 kernel yet, and I don't have time to mess around with the dev kernels, so I'm without any experimental data for you. From braun at net.in.tum.de Tue Nov 1 16:48:59 2011 From: braun at net.in.tum.de (Lothar Braun) Date: Wed, 2 Nov 2011 00:48:59 +0100 Subject: [Bro-Dev] snaplen and drops In-Reply-To: References: <20111027042458.GA87273@icir.org> <535469B6-200D-4940-A2E5-704C89457850@net.in.tum.de> <20111028033646.GA90146@icir.org> <20111028155635.GB12238@icir.org> Message-ID: <6655064A-DED1-47A5-B7AB-F12868211577@net.in.tum.de> Hi, On Oct 29, 2011, at 12:21 AM, Martin Holste wrote: > Glad you were able to sort this out. I use PF_RING exclusively for > packet capture, so I've not run into this before. > > In the future, AF_PACKET support would be a great addition to Bro and > would bring it closer to Snort and Suricata as far as acquisition. > It's got performance reasonably close to PF_RING without having to > download anything extra. I'm a bit puzzled. If I understand things correctly, libpcap-1.0.0 uses AF_PACKET by default (after checking that MMAP support is available in the running kernel). As far as I understand, AF_PACKET is the kernel socket infrastructure that allows to have a mmaped buffer between the kernel and userspace and a socket that can be polled/waited when no packets are stored in the buffer. Using a "new" libpcap with a modern kernel should already provide AF_PACKET support. Am I missing something? > However, you need to be running a 3.0 Linux > kernel to do software load-balancing, which is one of the reasons I > use PF_RING. Cool, I wasn't aware of load balancing features in the standard kernel. Did you do some experiments to compare the standard kernel load-balancing to the one provided by PF_RING? Best regards, Lothar -- Lothar Braun Chair for Network Architectures and Services (I8) Department of Informatics Technische Universit?t M?nchen Boltzmannstr. 3, 85748 Garching bei M?nchen, Germany Phone: +49 89 289-18010 Fax: +49 89 289-18033 E-mail: braun at net.in.tum.de From braun at net.in.tum.de Tue Nov 1 16:43:11 2011 From: braun at net.in.tum.de (Lothar Braun) Date: Wed, 2 Nov 2011 00:43:11 +0100 Subject: [Bro-Dev] snaplen and drops In-Reply-To: <20111028155635.GB12238@icir.org> References: <20111027042458.GA87273@icir.org> <535469B6-200D-4940-A2E5-704C89457850@net.in.tum.de> <20111028033646.GA90146@icir.org> <20111028155635.GB12238@icir.org> Message-ID: <20077DD7-6A50-4887-9DD4-D5C28DA2F17F@net.in.tum.de> Hi, On Oct 28, 2011, at 5:56 PM, Robin Sommer wrote: > Anyways, for Bro is probably makes most sense to address this as a > part of a larger piece we already have on our to-do list: overhauling > Bro's code for packet aquisition. It's in pretty bad shape right now: > (1) the main packet loop still works around problems with non-blocking > mode in older libpcap/OS versions; I would hope that's not necessary > anymore. (2), we don't have a nice interface for using other packet > sources than libpcap; we need an abstraction there. Snort has an abstraction layer called libdaq: http://vrt-blog.snort.org/2010/08/snort-29-essentials-daq.html I haven't had a look at it myself, so I cannot make a statement on whether its a good abstraction layer. But maybe it can be used in Bro, too. > And finally (3), > if we got an interface in to exploit further NIC-level features, like > load-balancing, that would be pretty cool. Yes, these new features might be very cool. However, if you rely on these hardware features, you might run into hardware-specifc problems. Some NICs seem to use a per-flow scheme for distributing traffic onto multiple queues. This can lead to problems if you use such NICs for distributing traffic to multiple Bro instances: Client and server traffic of a single TCP connection might be forwarded to different worker nodes. We therefore use software load-balancing for setups with multiple Bro worker nodes on a single machine. Best regards, Lothar -- Lothar Braun Chair for Network Architectures and Services (I8) Department of Informatics Technische Universit?t M?nchen Boltzmannstr. 3, 85748 Garching bei M?nchen, Germany Phone: +49 89 289-18010 Fax: +49 89 289-18033 E-mail: braun at net.in.tum.de From seth at icir.org Wed Nov 2 06:37:41 2011 From: seth at icir.org (Seth Hall) Date: Wed, 2 Nov 2011 09:37:41 -0400 Subject: [Bro-Dev] snaplen and drops In-Reply-To: <20077DD7-6A50-4887-9DD4-D5C28DA2F17F@net.in.tum.de> References: <20111027042458.GA87273@icir.org> <535469B6-200D-4940-A2E5-704C89457850@net.in.tum.de> <20111028033646.GA90146@icir.org> <20111028155635.GB12238@icir.org> <20077DD7-6A50-4887-9DD4-D5C28DA2F17F@net.in.tum.de> Message-ID: <73685948-1898-4A53-A3BF-E48814D2F794@icir.org> On Nov 1, 2011, at 7:43 PM, Lothar Braun wrote: > Snort has an abstraction layer called libdaq: > I haven't had a look at it myself, so I cannot make a statement on whether its a good abstraction layer. But maybe it can be used in Bro, too. Nope, it's GPL. I asked on their mailing list if they could relicense it as BSD right after they announced it a while back and I never heard back from anyone. > Some NICs seem to use a per-flow scheme for distributing traffic onto multiple queues. This can lead to problems if you use such NICs for distributing traffic to multiple Bro instances: Client and server traffic of a single TCP connection might be forwarded to different worker nodes. The fairly common RSS feature in NICs does this sort of round robin packet distribution across queues, but Intel's newer Flow Director feature on their high ends NICs does flow based load balancing across the queues so you actually get client and server traffic in the same queue. On Linux, the only way I know to take advantage of that is with TNAPI (from Luca Deri and the other NTOP guys). Lately I've been very impressed with Myricom's sniffer drivers which do the hardware based load balancing and direct memory injection. Their drivers work on FreeBSD and Linux which is an added benefit too. We need to make some small modifications to broctl to better support clustering with them, but they've been very problem free so far. Charging extra for special drivers seems a bit underhanded to me though (Myricom's sniffer drivers cost extra). > We therefore use software load-balancing for setups with multiple Bro worker nodes on a single machine. How are you doing this? PF_RING is also doing software based load balancing in the kernel, but it's actually slightly wrong because it includes the vlan-id as one of the tuples they balance on which can cause problems for network where each direction of traffic is in a different vlan. I filed a ticket with them to make the load balancing configurable though so hopefully that will be fixed in the next release of PF_RING. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Wed Nov 2 07:54:55 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 02 Nov 2011 14:54:55 -0000 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> Message-ID: <062.dd771abb767611e145e40da763615872@tracker.bro-ids.org> #388: Fix more compiler warnings ---------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Task | Status: assigned Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ---------------------+---------------------- Comment (by jsiwek): In [cec4600d2e16aadb9d24954f562f55edb0398a3d/bro]: {{{ #!CommitTicketReference repository="bro" revision="cec4600d2e16aadb9d24954f562f55edb0398a3d" Fixing compiler warnings (addresses #388) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 2 07:55:08 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 02 Nov 2011 14:55:08 -0000 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> Message-ID: <062.32385668d9dd7347c275f352fba644a1@tracker.bro-ids.org> #388: Fix more compiler warnings ---------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Task | Status: assigned Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ---------------------+---------------------- Comment (by jsiwek): In [8f2d9c71e4fcfaa5e2fb776de1519c056424b359/bro-aux]: {{{ #!CommitTicketReference repository="bro-aux" revision="8f2d9c71e4fcfaa5e2fb776de1519c056424b359" Fixing compiler warnings (addresses #388) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 2 07:55:20 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 02 Nov 2011 14:55:20 -0000 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> Message-ID: <062.36d9c4febc40955fed4b7da9c2dc0e44@tracker.bro-ids.org> #388: Fix more compiler warnings ---------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Task | Status: assigned Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ---------------------+---------------------- Comment (by jsiwek): In [32ce058bf7a1e15b98fed049546f8e823b8e52c0/broccoli]: {{{ #!CommitTicketReference repository="broccoli" revision="32ce058bf7a1e15b98fed049546f8e823b8e52c0" Fixing compiler warnings (addresses #388) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 2 07:55:37 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 02 Nov 2011 14:55:37 -0000 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> Message-ID: <062.d6f942c87e62d8a588a48754df8ec24b@tracker.bro-ids.org> #388: Fix more compiler warnings ---------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Task | Status: assigned Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ---------------------+---------------------- Comment (by jsiwek): In [d1e8a2a93bdd04b0ca9c3cb88a0e9730e3b2b72b/broccoli-python]: {{{ #!CommitTicketReference repository="broccoli-python" revision="d1e8a2a93bdd04b0ca9c3cb88a0e9730e3b2b72b" Fixing compiler warnings (addresses #388) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 2 07:55:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 02 Nov 2011 14:55:49 -0000 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> Message-ID: <062.654aa6d479b44ba45d0a98f33e266168@tracker.bro-ids.org> #388: Fix more compiler warnings ---------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Task | Status: assigned Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ---------------------+---------------------- Comment (by jsiwek): In [d321bda98f7f8f8ace2a22664e247f8d0e4b3509/broccoli-ruby]: {{{ #!CommitTicketReference repository="broccoli-ruby" revision="d321bda98f7f8f8ace2a22664e247f8d0e4b3509" Ignoring some SWIG warnings (addresses #388) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 2 07:56:15 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 02 Nov 2011 14:56:15 -0000 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> Message-ID: <062.3e1be3cc922972eaf22d49578a348bba@tracker.bro-ids.org> #388: Fix more compiler warnings ---------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Task | Status: assigned Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ---------------------+---------------------- Comment (by jsiwek): In [21b7978199a6f34a28fda3f858398126cc00dce7/pysubnettree]: {{{ #!CommitTicketReference repository="pysubnettree" revision="21b7978199a6f34a28fda3f858398126cc00dce7" Fixing compiler warnings (addresses #388) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 2 08:01:33 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 02 Nov 2011 15:01:33 -0000 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> Message-ID: <062.8cfb6d1dcfd17ee28b0f59d30a818c83@tracker.bro-ids.org> #388: Fix more compiler warnings ----------------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------------+---------------------- Changes (by jsiwek): * type: Task => Merge Request Comment: The commits above in the branch `topic/jsiwek/compiler-warnings` in `pysubnettree`, `broccoli-ruby`, `broccoli-python`, `broccoli`, `bro-aux`, and `bro` get rid of all the compile warnings for me on various platforms (Mac Lion, Debian 6, Centos 5/6, Fedora 15, Ubuntu 11.10... mixed x86_64/x86). -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 2 08:08:56 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 02 Nov 2011 15:08:56 -0000 Subject: [Bro-Dev] #657: Change bro-cut for general timestamp conversion. Message-ID: <046.9668fa90eb8243c2b8a9c55df2a6a329@tracker.bro-ids.org> #657: Change bro-cut for general timestamp conversion. -----------------------------+------------------------ Reporter: seth | Owner: robin Type: Feature Request | Status: new Priority: Normal | Milestone: Bro2.0 Component: bro-aux | Version: git/master Keywords: | -----------------------------+------------------------ It should be possible to not give bro-cut any fields to cut out to get everything passed through but still convert timestamps. This is particularly relevant due to the removal of cf from the base distribution and especially makes sense in light of logs that are self descriptive so not limited to timestamps in the first column. This should work: {{{ cat conn.log | bro-cut -d }}} Since bro-cut is basically just an awk program anyway, this should work too (it would be relatively easy to add, right?): {{{ cat conn.log | bro-cut -c id.orig_h=1.2.3.4 }}} I'm using "-c" above as "condition". This may be too much for now though, the all fields timestamp thing is important though (due to lack of cf). -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Wed Nov 2 08:09:40 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 2 Nov 2011 08:09:40 -0700 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <062.8cfb6d1dcfd17ee28b0f59d30a818c83@tracker.bro-ids.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> <062.8cfb6d1dcfd17ee28b0f59d30a818c83@tracker.bro-ids.org> Message-ID: <20111102150940.GA83396@icir.org> On Wed, Nov 02, 2011 at 15:01 -0000, you wrote: > Changes (by jsiwek): > The commits above in the branch `topic/jsiwek/compiler-warnings` in > `pysubnettree`, `broccoli-ruby`, `broccoli-python`, `broccoli`, `bro-aux`, > and `bro` get rid of all the compile warnings for me on various platforms > (Mac Lion, Debian 6, Centos 5/6, Fedora 15, Ubuntu 11.10... mixed > x86_64/x86). Very cool, thanks for that work! Is there a way to integrate checking for warnings into the NMI test building? I don't want to generally turn compiler warnings into errors as they can appear quite suddently. But having a way to notice and flag them would be nice, perhaps is a separate NMI mail that just greps through the output for any warnings? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Wed Nov 2 08:12:54 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 02 Nov 2011 15:12:54 -0000 Subject: [Bro-Dev] #658: Broctl check doesn't account for cluster layout or config changes Message-ID: <046.33069865e6009f5d564323acf1468ca8@tracker.bro-ids.org> #658: Broctl check doesn't account for cluster layout or config changes ------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: Keywords: | ------------------------+-------------------- There are two parts to the "install" command in broctl. Copying files and creating the autogenerated scripts. The "check" command takes care of the copied files by adjusting the BROPATH, but it doesn't account for configuration changes in node.cfg or broctl.cfg. The "check" command needs to generate temporary versions of the autogenerated files to keep the process of "check -> install -> restart" the same no matter where changes were made. -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Wed Nov 2 08:18:48 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 2 Nov 2011 10:18:48 -0500 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <20111102150940.GA83396@icir.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> <062.8cfb6d1dcfd17ee28b0f59d30a818c83@tracker.bro-ids.org> <20111102150940.GA83396@icir.org> Message-ID: <9C07BD99-72D7-4CEA-A92E-8CF64C46B7EF@ncsa.illinois.edu> > Is there a way to integrate checking for warnings into the NMI test > building? Yeah, I think that warnings will always be printed on stderr, so we could treat any output there as a build "error". Or we could also grep for "warning" case insensitive in the stdout from make like you said. I'll create a ticket assigned to Daniel for adding that. - Jon From bro at tracker.bro-ids.org Wed Nov 2 08:23:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 02 Nov 2011 15:23:28 -0000 Subject: [Bro-Dev] #659: NMI builds check for compiler warnings Message-ID: <048.98a7f5e7f513339f0f639eb6aee714a6@tracker.bro-ids.org> #659: NMI builds check for compiler warnings -----------------------------+------------------------ Reporter: jsiwek | Owner: dnthayer Type: Feature Request | Status: new Priority: Low | Milestone: Bro2.0 Component: Bro | Version: git/master Keywords: | -----------------------------+------------------------ The NMI build tests should check for compiler warnings. They should always be observable on stderr, but it shouldn't hurt to also grep stdout for "warning" case-insensitive and then trigger an NMI error if anything is found. -- Ticket URL: Bro Tracker Bro Issue Tracker From hlin33 at illinois.edu Wed Nov 2 08:51:13 2011 From: hlin33 at illinois.edu (Hui Lin (Hugo) ) Date: Wed, 2 Nov 2011 10:51:13 -0500 Subject: [Bro-Dev] Hui Lin_How to disable some default log option Message-ID: Hi, I have update with master branch of Bro (still Bro 1.6). But I find that currently, by default, Bro will generate conn.log, noitice_policy.log and packet_filter.log without indicating any bro policy script. I am wondering is there any way that I can disable them. Best, Hui -- Hui Lin Research Assistant DEPEND Research Group, ECE Department University of Illinois at Urbana-Champaign hlin33 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20111102/002f0394/attachment.html From seth at icir.org Wed Nov 2 09:00:36 2011 From: seth at icir.org (Seth Hall) Date: Wed, 2 Nov 2011 12:00:36 -0400 Subject: [Bro-Dev] Hui Lin_How to disable some default log option In-Reply-To: References: Message-ID: <552CA133-3355-4A79-B582-025F76C876D9@icir.org> On Nov 2, 2011, at 11:51 AM, Hui Lin (Hugo) wrote: > I have update with master branch of Bro (still Bro 1.6). That's actually called 2.0, there isn't really a 1.6. ;) > But I find that currently, by default, Bro will generate conn.log, noitice_policy.log and packet_filter.log without indicating any bro policy script. I am wondering is there any way that I can disable them. Search for "disable_stream" in the logging framework documentation: http://www.bro-ids.org/documentation-beta/logging.bro.html The streams you're looking to disable are: Conn::LOG Notice::POLICY_LOG PacketFilter::LOG .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From gregor at icir.org Wed Nov 2 09:02:34 2011 From: gregor at icir.org (Gregor Maier) Date: Wed, 02 Nov 2011 17:02:34 +0100 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <20111102150940.GA83396@icir.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> <062.8cfb6d1dcfd17ee28b0f59d30a818c83@tracker.bro-ids.org> <20111102150940.GA83396@icir.org> Message-ID: <4EB1699A.70002@icir.org> > Is there a way to integrate checking for warnings into the NMI test > building? I don't want to generally turn compiler warnings into > errors as they can appear quite suddently. But having a way to notice > and flag them would be nice, perhaps is a separate NMI mail that just > greps through the output for any warnings? One question would be whether it's acceptable to add a configure option that compiles with -Werror and then do the NMI builds with that..... cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From jsiwek at ncsa.illinois.edu Wed Nov 2 09:12:51 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 2 Nov 2011 11:12:51 -0500 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <4EB1699A.70002@icir.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> <062.8cfb6d1dcfd17ee28b0f59d30a818c83@tracker.bro-ids.org> <20111102150940.GA83396@icir.org> <4EB1699A.70002@icir.org> Message-ID: > One question would be whether it's acceptable to add a configure option > that compiles with -Werror and then do the NMI builds with that?. I think what we want is for the build to complete even if there's warnings so that the tests can still run. i.e. A given NMI job submission should tell us the most information as we can get from it. - Jon From bro at tracker.bro-ids.org Wed Nov 2 09:29:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 02 Nov 2011 16:29:36 -0000 Subject: [Bro-Dev] #645: Figure out where sendmail is In-Reply-To: <047.30bfd70cef14df8f513f2d5595dceaae@tracker.bro-ids.org> References: <047.30bfd70cef14df8f513f2d5595dceaae@tracker.bro-ids.org> Message-ID: <062.1cb2f3370da5a90771a477b6e4e12078@tracker.bro-ids.org> #645: Figure out where sendmail is -------------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Problem | Status: assigned Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: Resolution: | Keywords: -------------------------+---------------------- Comment (by jsiwek): I added the configure-time check in fastpath [5478f4fdecad980c8821b8e2c7c3b2cbb9908917/broctl], but typo'd the trac- integration commit message. I had settled on printing a warning if no sendmail is found, but proceeding with the build since running broctl without sendmail is not an erroneous configuration. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 2 09:48:57 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 02 Nov 2011 16:48:57 -0000 Subject: [Bro-Dev] #624: Bro won't shutdown if not seeing packets In-Reply-To: <046.06582d3d3a1ee7e1010ccc163ac53a7f@tracker.bro-ids.org> References: <046.06582d3d3a1ee7e1010ccc163ac53a7f@tracker.bro-ids.org> Message-ID: <061.7d3e22d052fa80db7fa1b8eccbd4fd2a@tracker.bro-ids.org> #624: Bro won't shutdown if not seeing packets ---------------------------+-------------------- Reporter: seth | Owner: jsiwek Type: Problem | Status: closed Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: Works for Me | Keywords: ---------------------------+-------------------- Changes (by jsiwek): * status: assigned => closed * resolution: => Works for Me Comment: I couldn't reproduce this with a pf_ring enabled virtual cluster with one node listening on a dummy interface. This may have been related to the capstats bug now fixed, so I'm going to close it. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 2 13:31:58 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 02 Nov 2011 20:31:58 -0000 Subject: [Bro-Dev] #660: Initializing a table with a record as an index does not work Message-ID: <048.67b15571cb4de0bd789177060fea4dc2@tracker.bro-ids.org> #660: Initializing a table with a record as an index does not work --------------------+--------------------- Reporter: amannb | Type: Problem Status: new | Priority: Normal Milestone: | Component: Bro Version: | Keywords: --------------------+--------------------- The following code {{{ type irecord: record { resp_h: addr; }; global my_table: table[irecord] of int = { [[$resp_h=1.2.3.4]] = 12, }; }}} results in an error: {{{ error in ./bug2.bro, line 7: illegal initializer ([[$resp_h=1.2.3.4]] = 12) }}} It probably should return an initialized table. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 2 14:15:04 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 02 Nov 2011 21:15:04 -0000 Subject: [Bro-Dev] #660: Initializing a table with a record as an index does not work In-Reply-To: <048.67b15571cb4de0bd789177060fea4dc2@tracker.bro-ids.org> References: <048.67b15571cb4de0bd789177060fea4dc2@tracker.bro-ids.org> Message-ID: <063.3a29b807bae44296a1e7336242d08c61@tracker.bro-ids.org> #660: Initializing a table with a record as an index does not work ----------------------+----------------- Reporter: amannb | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: Resolution: | Keywords: ----------------------+----------------- Comment (by seth): > global my_table: table[irecord] of int = { Using a record as an index on a table or set is usually not a good idea. Lookups may not happen as you expect if you are using a different record instance to index into the table or set since Bro isn't doing value equality but actually looking for the same record. -- Ticket URL: Bro Tracker Bro Issue Tracker From jones at tacc.utexas.edu Wed Nov 2 14:29:43 2011 From: jones at tacc.utexas.edu (William Jones) Date: Wed, 2 Nov 2011 21:29:43 +0000 Subject: [Bro-Dev] #656: Add stats.bro back In-Reply-To: <047.2d9bdf13c6cee04531b4645e09da8cb7@tracker.bro-ids.org> References: <047.2d9bdf13c6cee04531b4645e09da8cb7@tracker.bro-ids.org> Message-ID: Attached is a replacement script for stats.bro that works with Bro-beta 2.0. It is a small rewrite of the original. Use if in bro if you think it's useful. I am still learning bro scripting so I wouldn't mind any feed backup on the script. Bill Jones -----Original Message----- From: bro-dev-bounces at bro-ids.org [mailto:bro-dev-bounces at bro-ids.org] On Behalf Of Bro Tracker Sent: Tuesday, November 01, 2011 9:48 AM To: robin at icir.org Cc: bro-dev at bro-ids.org Subject: [Bro-Dev] #656: Add stats.bro back #656: Add stats.bro back ------------------------+-------------------- Reporter: robin | Type: Task Status: new | Priority: Normal Milestone: Bro2.0 | Component: Bro Version: git/master | ------------------------+-------------------- That didn't make it into the new set of scripts so far, but should be pretty straight-forward to put back in place. -- Ticket URL: Bro Tracker Bro Issue Tracker _______________________________________________ bro-dev mailing list bro-dev at bro-ids.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: stats.bro Type: application/octet-stream Size: 1952 bytes Desc: stats.bro Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20111102/a15f4b97/attachment.obj From bro at tracker.bro-ids.org Wed Nov 2 14:55:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 02 Nov 2011 21:55:28 -0000 Subject: [Bro-Dev] #660: Initializing a table with a record as an index does not work In-Reply-To: <048.67b15571cb4de0bd789177060fea4dc2@tracker.bro-ids.org> References: <048.67b15571cb4de0bd789177060fea4dc2@tracker.bro-ids.org> Message-ID: <063.52a69b30953c9be08c806976beb5cf10@tracker.bro-ids.org> #660: Initializing a table with a record as an index does not work ----------------------+----------------- Reporter: amannb | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: Resolution: | Keywords: ----------------------+----------------- Comment (by robin): On Wed, Nov 02, 2011 at 21:15 -0000, you wrote: > Using a record as an index on a table or set is usually not a good idea. > Lookups may not happen as you expect if you are using a different record > instance to index into the table or set since Bro isn't doing value > equality but actually looking for the same record. Are you sure? This works as I would expect it: {{{ type r: record { a: string; }; global foo: table[r] of count; global a: r = [$a="foo"]; global b: r = [$a="foo"]; global c: r = [$a="bar"]; foo[a] = 1; print a in foo; print b in foo; print c in foo; }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From gc355804 at ohio.edu Wed Nov 2 16:03:10 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Wed, 02 Nov 2011 19:03:10 -0400 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> <062.8cfb6d1dcfd17ee28b0f59d30a818c83@tracker.bro-ids.org> <20111102150940.GA83396@icir.org> <4EB1699A.70002@icir.org> Message-ID: <4EB1CC2E.7010807@ohio.edu> Could we turn a rebuild of the source code with the -Werror option into a btest case, rather than actually using it for the initial build? --Gilbert On 11/2/2011 12:12 PM, Jonathan Siwek wrote: >> One question would be whether it's acceptable to add a configure option >> that compiles with -Werror and then do the NMI builds with that?. > I think what we want is for the build to complete even if there's warnings so that the tests can still run. i.e. A given NMI job submission should tell us the most information as we can get from it. > > - Jon > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From jsiwek at ncsa.illinois.edu Wed Nov 2 17:14:53 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 2 Nov 2011 19:14:53 -0500 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <4EB1CC2E.7010807@ohio.edu> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> <062.8cfb6d1dcfd17ee28b0f59d30a818c83@tracker.bro-ids.org> <20111102150940.GA83396@icir.org> <4EB1699A.70002@icir.org> <4EB1CC2E.7010807@ohio.edu> Message-ID: <6A33E935-AAC3-43C3-A027-8D4AA7E6A0DD@ncsa.illinois.edu> > Could we turn a rebuild of the source code with the -Werror option into > a btest case, rather than actually using it for the initial build? That's a neat idea, however, any use of -Werror is going to abort the build prematurely, right? So we'll only be capturing the details of the first instance of a warning being emitted if we rely on -Werror, when I think it's more useful if NMI could give us all the warnings that were emitted. But I'm still open to adding an additional btest that uses -Werror that would be useful at getting developers to think twice about filing a merge request for code that breaks tests because of warnings. - Jon From gc355804 at ohio.edu Wed Nov 2 17:37:45 2011 From: gc355804 at ohio.edu (G. Clark) Date: Wed, 02 Nov 2011 20:37:45 -0400 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <6A33E935-AAC3-43C3-A027-8D4AA7E6A0DD@ncsa.illinois.edu> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> <062.8cfb6d1dcfd17ee28b0f59d30a818c83@tracker.bro-ids.org> <20111102150940.GA83396@icir.org> <4EB1699A.70002@icir.org> <4EB1CC2E.7010807@ohio.edu> <6A33E935-AAC3-43C3-A027-8D4AA7E6A0DD@ncsa.illinois.edu> Message-ID: <4EB1E259.4030701@ohio.edu> On 11/2/11 8:14 PM, Jonathan Siwek wrote: > That's a neat idea, however, any use of -Werror is going to abort the build prematurely, right? So we'll only be capturing the details of the first instance of a warning being emitted if we rely on -Werror, when I think it's more useful if NMI could give us all the warnings that were emitted. True. How about redirecting make output to a log and running a script against the output? Any warning: output fails the test, but we still get the entirety of the output this way. --Gilbert -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6010 bytes Desc: S/MIME Cryptographic Signature Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20111102/1124fe0a/attachment-0001.bin From bro at tracker.bro-ids.org Wed Nov 2 17:55:38 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 03 Nov 2011 00:55:38 -0000 Subject: [Bro-Dev] #660: Initializing a table with a record as an index does not work In-Reply-To: <048.67b15571cb4de0bd789177060fea4dc2@tracker.bro-ids.org> References: <048.67b15571cb4de0bd789177060fea4dc2@tracker.bro-ids.org> Message-ID: <063.846c85767c56a8dca86d107269d38c6f@tracker.bro-ids.org> #660: Initializing a table with a record as an index does not work ----------------------+----------------- Reporter: amannb | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: Resolution: | Keywords: ----------------------+----------------- Comment (by seth): > Are you sure? This works as I would expect it: Damn, you're right. We should talk about finding some way to rethink record and table initializers eventually. There are too many inconsistencies with how they are done currently. -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Wed Nov 2 18:06:05 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 2 Nov 2011 20:06:05 -0500 (CDT) Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <4EB1E259.4030701@ohio.edu> Message-ID: <5527271.1733.1320282365575.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > True. How about redirecting make output to a log and running a script > against the output? Any warning: output fails the test, but we still > get the entirety of the output this way. Yeah, that's the way I was imagining. Does anyone like the idea of this being a regular btest versus something that only runs on NMI? I'm not sure how much benefit is derived from the former at the expense of increasing the testing time by a few minutes. +Jon From seth at icir.org Wed Nov 2 18:13:35 2011 From: seth at icir.org (Seth Hall) Date: Wed, 2 Nov 2011 21:13:35 -0400 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <5527271.1733.1320282365575.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <5527271.1733.1320282365575.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <8821655A-5824-4691-99A5-DCB223C357C7@icir.org> On Nov 2, 2011, at 9:06 PM, Jonathan Siwek wrote: > Yeah, that's the way I was imagining. Does anyone like the idea of this being a regular btest versus something that only runs on NMI? I'm not sure how much benefit is derived from the former at the expense of increasing the testing time by a few minutes. I think that makes sense. We'd at least get a chance to see if we introduces any warnings locally before committing where they may be exposed on other platforms on NMI. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From braun at net.in.tum.de Thu Nov 3 05:58:51 2011 From: braun at net.in.tum.de (Lothar Braun) Date: Thu, 3 Nov 2011 13:58:51 +0100 Subject: [Bro-Dev] snaplen and drops In-Reply-To: References: <20111027042458.GA87273@icir.org> <535469B6-200D-4940-A2E5-704C89457850@net.in.tum.de> <20111028033646.GA90146@icir.org> <20111028155635.GB12238@icir.org> <6655064A-DED1-47A5-B7AB-F12868211577@net.in.tum.de> Message-ID: Hi, On Nov 2, 2011, at 2:51 AM, Martin Holste wrote: >> I'm a bit puzzled. If I understand things correctly, libpcap-1.0.0 uses AF_PACKET by default (after checking that MMAP support is available in the running kernel). >> > > I don't think that's how it works, but I'm not a kernel-hacking guru. > The reason I'm pretty sure it doesn't work that way is that both Snort > and Suricata IDS include separate data acquisition code for libpcap > and af_packet, which is nonsensical if you can get af_packet via > libpcap natively. I did a bit of Googling and cannot find anything > definitive one way or the other. I had a quick look at the libpcap (1.2.0) and the libdaq (0.6.2) code. It seems to me that both of them perform basically the same steps for packet acquisition. Both create a socket PF_PACKET socket, both request a shared memory area on the capturing rx_ring. And both perform similar operations during packet acquisition: while (running) { if (packet_in_buffer) { consume_packet(); } else { poll(socket); } } So if you use libpcap >= 1.0.0, you should have AF_PACKET support by default. Snort/Suricata probably implemented separate AF_PACKET support for systems that ship libpcap < 1.0.0. Best regards, Lothar -- Lothar Braun Chair for Network Architectures and Services (I8) Department of Informatics Technische Universit?t M?nchen Boltzmannstr. 3, 85748 Garching bei M?nchen, Germany Phone: +49 89 289-18010 Fax: +49 89 289-18033 E-mail: braun at net.in.tum.de From braun at net.in.tum.de Thu Nov 3 06:05:00 2011 From: braun at net.in.tum.de (Lothar Braun) Date: Thu, 3 Nov 2011 14:05:00 +0100 Subject: [Bro-Dev] snaplen and drops In-Reply-To: <73685948-1898-4A53-A3BF-E48814D2F794@icir.org> References: <20111027042458.GA87273@icir.org> <535469B6-200D-4940-A2E5-704C89457850@net.in.tum.de> <20111028033646.GA90146@icir.org> <20111028155635.GB12238@icir.org> <20077DD7-6A50-4887-9DD4-D5C28DA2F17F@net.in.tum.de> <73685948-1898-4A53-A3BF-E48814D2F794@icir.org> Message-ID: <3961C8A1-4840-4E8D-8828-7881AC79A9BC@net.in.tum.de> Hi, On Nov 2, 2011, at 2:37 PM, Seth Hall wrote: >> Some NICs seem to use a per-flow scheme for distributing traffic onto multiple queues. This can lead to problems if you use such NICs for distributing traffic to multiple Bro instances: Client and server traffic of a single TCP connection might be forwarded to different worker nodes. > > The fairly common RSS feature in NICs does this sort of round robin packet distribution across queues, but Intel's newer Flow Director feature on their high ends NICs does flow based load balancing across the queues so you actually get client and server traffic in the same queue. On Linux, the only way I know to take advantage of that is with TNAPI (from Luca Deri and the other NTOP guys). we use an Intel card (82598 Based PCI-E 10) that supports flow-based distribution across queues (RSS). And flow-based on this chipsset really means flow-based. Each flow seems to be hashed according to some hash function that works on: H(srcIP, srcPort, dstIP, dstPort, proto) instead of something like H(srcIP + srcPort + dstIP + dstPort + proto) which would be bi-directional. I just checked with one of our setups that employs 8 RX queues with PF_RING + TNAPI. This setup distributes client and server side traffic from several connections onto different queues. Since I don't have a 82599-based card, I don't know if they changed this behavior. However, from a network card engineers' points of view, it seems to be perfectly fine to not map bi-flows to different queues but split the directions of the traffic on a per-flow base: If you operate a NIC in a server for normal network communication and not traffic monitoring, which is probably the primary use case for NICs, you will have client traffic on the RX queues and server traffic on the TX queues. So both directions can be hashed independently. Anyways, the original point I was trying to make is: If you employ the hardware features of current NICs directly, you might run into hardware-specific issues. Such as different chipsets (or even worse: different chipset revisions), that oppose different behavior. > Lately I've been very impressed with Myricom's sniffer drivers which do the hardware based load balancing and direct memory injection. Is this similar to the DNA driver that has been done by Luca Deri? >> We therefore use software load-balancing for setups with multiple Bro worker nodes on a single machine. > > > How are you doing this? PF_RING is also doing software based load balancing in the kernel, but it's actually slightly wrong because it includes the vlan-id as one of the tuples they balance on which can cause problems for network where each direction of traffic is in a different vlan. I filed a ticket with them to make the load balancing configurable though so hopefully that will be fixed in the next release of PF_RING. Yes, we use this PF_RING feature for load-balancing. I haven't had any problems with the VLAN problem that you describe, as we do not have any PF_RING setups that employ multiple VLANs. But yes, this behavior is definitely a bug, and should be fixed. Best regards, Lothar -- Lothar Braun Chair for Network Architectures and Services (I8) Department of Informatics Technische Universit?t M?nchen Boltzmannstr. 3, 85748 Garching bei M?nchen, Germany Phone: +49 89 289-18010 Fax: +49 89 289-18033 E-mail: braun at net.in.tum.de From seth at icir.org Thu Nov 3 06:40:11 2011 From: seth at icir.org (Seth Hall) Date: Thu, 3 Nov 2011 09:40:11 -0400 Subject: [Bro-Dev] snaplen and drops In-Reply-To: <3961C8A1-4840-4E8D-8828-7881AC79A9BC@net.in.tum.de> References: <20111027042458.GA87273@icir.org> <535469B6-200D-4940-A2E5-704C89457850@net.in.tum.de> <20111028033646.GA90146@icir.org> <20111028155635.GB12238@icir.org> <20077DD7-6A50-4887-9DD4-D5C28DA2F17F@net.in.tum.de> <73685948-1898-4A53-A3BF-E48814D2F794@icir.org> <3961C8A1-4840-4E8D-8828-7881AC79A9BC@net.in.tum.de> Message-ID: <0F6C30B7-7C93-4F9B-BFC7-F190BA079CA7@icir.org> On Nov 3, 2011, at 9:05 AM, Lothar Braun wrote: > H(srcIP, srcPort, dstIP, dstPort, proto) > > instead of something like > > H(srcIP + srcPort + dstIP + dstPort + proto) Ugh.. Well that's annoying. So, I guess the best option currently with a commodity NIC is to still do the load balancing on the CPU? (with PF_RING or AF_PACKET since that seems to support load balancing now too we don't support it yet though) >> Lately I've been very impressed with Myricom's sniffer drivers which do the hardware based load balancing and direct memory injection. > > Is this similar to the DNA driver that has been done by Luca Deri? In a way since it does include the direct memory injection like the DNA drivers but it also does the multiqueue bidirectional flow based load balancing. I've just been very impressed with the myricom drivers because they're easy to install and start using. We need to make some small changes to broctl to properly support doing the load balancing with the myricom sniffer drivers, but nothing major. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Thu Nov 3 06:42:10 2011 From: seth at icir.org (Seth Hall) Date: Thu, 3 Nov 2011 09:42:10 -0400 Subject: [Bro-Dev] snaplen and drops In-Reply-To: References: <20111027042458.GA87273@icir.org> <535469B6-200D-4940-A2E5-704C89457850@net.in.tum.de> <20111028033646.GA90146@icir.org> <20111028155635.GB12238@icir.org> <6655064A-DED1-47A5-B7AB-F12868211577@net.in.tum.de> Message-ID: <198E3AD7-715B-461B-9EDA-6B110B20888C@icir.org> On Nov 3, 2011, at 8:58 AM, Lothar Braun wrote: > I had a quick look at the libpcap (1.2.0) and the libdaq (0.6.2) code. It seems to me that both of them perform basically the same steps for packet acquisition. > > Both create a socket PF_PACKET socket, both request a shared memory area on the capturing rx_ring. And both perform similar operations during packet acquisition: Did you look int the os-daq-modules/daq_afpacket.c file? DAQ implements the AF_PACKET support there. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From mcholste at gmail.com Thu Nov 3 06:51:05 2011 From: mcholste at gmail.com (Martin Holste) Date: Thu, 3 Nov 2011 08:51:05 -0500 Subject: [Bro-Dev] snaplen and drops In-Reply-To: References: <20111027042458.GA87273@icir.org> <535469B6-200D-4940-A2E5-704C89457850@net.in.tum.de> <20111028033646.GA90146@icir.org> <20111028155635.GB12238@icir.org> <6655064A-DED1-47A5-B7AB-F12868211577@net.in.tum.de> Message-ID: > So if you use libpcap >= 1.0.0, you should have AF_PACKET support by default. Snort/Suricata probably implemented separate AF_PACKET support for systems that ship libpcap < 1.0.0. I've used pcap > 1.0 and had much worse performance than AF_PACKET, so I'd be willing to bet that IRQ CPU utilization is higher with pcap and AF_PACKET does a polling mechanism to decrease its IRQ overhead. I can't speak to the mmap techniques and whether or not they differ, but IRQ alone would be enough to make a noticeable difference. From bro at tracker.bro-ids.org Thu Nov 3 09:07:27 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 03 Nov 2011 16:07:27 -0000 Subject: [Bro-Dev] #660: Initializing a table with a record as an index does not work In-Reply-To: <048.67b15571cb4de0bd789177060fea4dc2@tracker.bro-ids.org> References: <048.67b15571cb4de0bd789177060fea4dc2@tracker.bro-ids.org> Message-ID: <063.e256821399482848d32dd3ed27fe2b9d@tracker.bro-ids.org> #660: Initializing a table with a record as an index does not work ----------------------+---------------------- Reporter: amannb | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: Resolution: | Keywords: language ----------------------+---------------------- Changes (by robin): * keywords: => language Comment: Yeah, and using a two-element record for ``r`` gives me parser error. :-( -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Thu Nov 3 09:10:11 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 3 Nov 2011 09:10:11 -0700 Subject: [Bro-Dev] #656: Add stats.bro back In-Reply-To: References: <047.2d9bdf13c6cee04531b4645e09da8cb7@tracker.bro-ids.org> Message-ID: <20111103161011.GD42624@icir.org> On Wed, Nov 02, 2011 at 21:29 +0000, you wrote: > Attached is a replacement script for stats.bro that works with Bro-beta 2.0. Thanks! I've added it to the ticket. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Thu Nov 3 09:24:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 03 Nov 2011 16:24:36 -0000 Subject: [Bro-Dev] #661: Crash in host name lookup Message-ID: <047.b51474059b3622f2b1778a88ef2d7190@tracker.bro-ids.org> #661: Crash in host name lookup ---------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro2.0 Component: Bro | Version: Keywords: | ---------------------+-------------------- The following script gives me a glibc memory corruption error right at startup: {{{ const foo: set[addr] = { google.com }; }}} {{{ bro ./a.bro *** glibc detected *** bro: double free or corruption (out): 0x000000000147fca0 *** ======= Backtrace: ========= /lib64/libc.so.6[0x311547703a] bro(_ZN7DNS_Mgr7ResolveEv+0x1ff)[0x52374f] bro(_ZN7DNS_Mgr10LookupHostEPKc+0x16b)[0x523b8b] bro(_Z6brolexv+0x1c46)[0x4f1786] bro(_Z7yyparsev+0xa28)[0x4dfc78] bro(main+0x872)[0x4a4062] /lib64/libc.so.6(__libc_start_main+0xed)[0x311542139d] bro[0x4a81a1] }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Thu Nov 3 09:44:01 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 3 Nov 2011 09:44:01 -0700 Subject: [Bro-Dev] Hui Lin_How to disable some default log option In-Reply-To: <552CA133-3355-4A79-B582-025F76C876D9@icir.org> References: <552CA133-3355-4A79-B582-025F76C876D9@icir.org> Message-ID: <20111103164401.GJ42624@icir.org> On Wed, Nov 02, 2011 at 12:00 -0400, you wrote: > > But I find that currently, by default, Bro will generate conn.log, > > noitice_policy.log and packet_filter.log without indicating any bro > > policy script. I am wondering is there any way that I can disable > > them. > > Search for "disable_stream" in the logging framework documentation: Or, depending on what you want to do, enable "bare mode" with -b, which won't load any default scripts. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Thu Nov 3 10:01:33 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 03 Nov 2011 17:01:33 -0000 Subject: [Bro-Dev] #658: Broctl check doesn't account for cluster layout or config changes In-Reply-To: <046.33069865e6009f5d564323acf1468ca8@tracker.bro-ids.org> References: <046.33069865e6009f5d564323acf1468ca8@tracker.bro-ids.org> Message-ID: <061.2172a2435a9081903db58e25396f686e@tracker.bro-ids.org> #658: Broctl check doesn't account for cluster layout or config changes -------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: Resolution: | Keywords: -------------------------+-------------------- Comment (by robin): On Wed, Nov 02, 2011 at 15:12 -0000, you wrote: > The "check" command needs to generate temporary versions of the > autogenerated files to keep the process of "check -> install -> restart" > the same no matter where changes were made. Another issue (regression) in the same context: BroControl doesn't copy the site policy files somewhere else anymore to then run from there. According to loaded_scripts.log, it loads the original files instead. That means that when I edit, e.g., local.bro, the next restart will pick up the changes even if I don't do an install in between. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Thu Nov 3 10:07:05 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 3 Nov 2011 10:07:05 -0700 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <8821655A-5824-4691-99A5-DCB223C357C7@icir.org> References: <5527271.1733.1320282365575.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <8821655A-5824-4691-99A5-DCB223C357C7@icir.org> Message-ID: <20111103170705.GM42624@icir.org> On Wed, Nov 02, 2011 at 21:13 -0400, you wrote: > > Yeah, that's the way I was imagining. Does anyone like the idea of > > this being a regular btest versus something that only runs on NMI? > > I'm not sure how much benefit is derived from the former at the > > expense of increasing the testing time by a few minutes. > > > I think that makes sense. We'd at least get a chance to see if we > introduces any warnings locally before committing where they may be > exposed on other platforms on NMI. Checking locally is good, but I'm also concerned about increasing testing time for every btest run. How about making it an option? We can use @test-requires to test some preconditionn, like a set environment variable. Alternatively, I have a btest patch pending that introduces test groups. With that, we could make a group "long-tests" for tests that take a while to run. btest would then just need another option to not run certain groups by default (the patch already has an option to *only* run certain groups). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Thu Nov 3 10:18:11 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 03 Nov 2011 17:18:11 -0000 Subject: [Bro-Dev] #659: NMI builds check for compiler warnings In-Reply-To: <048.98a7f5e7f513339f0f639eb6aee714a6@tracker.bro-ids.org> References: <048.98a7f5e7f513339f0f639eb6aee714a6@tracker.bro-ids.org> Message-ID: <063.6fea43f9e267043c4a38a46d82774d0c@tracker.bro-ids.org> #659: NMI builds check for compiler warnings ------------------------------+------------------------ Reporter: jsiwek | Owner: dnthayer Type: Feature Request | Status: new Priority: Low | Milestone: Bro2.0 Component: Bro | Version: git/master Resolution: | Keywords: nmi ------------------------------+------------------------ Changes (by jsiwek): * keywords: => nmi Comment: Let's make this check for compiler warning output a btest that runs only in certain conditions e.g. @test-requires that checks for an environment variable or a btest group (pending patch from Robin). That way it will always run permanently on NMI, but we'll also have the option to easily run it locally. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Nov 3 10:41:46 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 03 Nov 2011 17:41:46 -0000 Subject: [Bro-Dev] #662: Patch to make BroControl Python 2.4 friendly Message-ID: <047.74c9f68692c0cdac77f6fb2c362eb07c@tracker.bro-ids.org> #662: Patch to make BroControl Python 2.4 friendly ------------------------+------------------------ Reporter: robin | Type: Patch Status: new | Priority: Normal Milestone: 2.0 | Component: BroControl Version: git/master | ------------------------+------------------------ Patch from Bill Jones. Still not sure we should make 2.4 officially supported, but applying the patch so that it will still work can't hurt it seems. [attachment:"python.patch"] -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Nov 3 15:26:03 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 03 Nov 2011 22:26:03 -0000 Subject: [Bro-Dev] #645: Figure out where sendmail is In-Reply-To: <047.30bfd70cef14df8f513f2d5595dceaae@tracker.bro-ids.org> References: <047.30bfd70cef14df8f513f2d5595dceaae@tracker.bro-ids.org> Message-ID: <062.848a7997e65604a0fca6b95af846e346@tracker.bro-ids.org> #645: Figure out where sendmail is -------------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Problem | Status: assigned Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: Resolution: | Keywords: -------------------------+---------------------- Comment (by robin): I'm merging this in, but I was hoping to completely remove the sendmail option from the default broctl.cfg. Can we preset configure-time options in options.py? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Nov 3 15:33:58 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 03 Nov 2011 22:33:58 -0000 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> Message-ID: <062.07c43f111c5a0855e279dcc3c907ec4a@tracker.bro-ids.org> #388: Fix more compiler warnings ----------------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------------+---------------------- Comment (by robin): In [e23c4092a6b8a2ec6161537dd45205f12b432b4a/broccoli-python]: {{{ #!CommitTicketReference repository="broccoli-python" revision="e23c4092a6b8a2ec6161537dd45205f12b432b4a" Merge remote-tracking branch 'origin/topic/jsiwek/compiler-warnings' * origin/topic/jsiwek/compiler-warnings: Fixing compiler warnings (addresses #388) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Nov 3 15:34:11 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 03 Nov 2011 22:34:11 -0000 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> Message-ID: <062.3207f5d6a66a65d07a1c5bb85673fe50@tracker.bro-ids.org> #388: Fix more compiler warnings ----------------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------------+---------------------- Comment (by robin): In [97abe472bb883a1e108602b37a59f03536283e33/pysubnettree]: {{{ #!CommitTicketReference repository="pysubnettree" revision="97abe472bb883a1e108602b37a59f03536283e33" Merge remote-tracking branch 'origin/topic/jsiwek/compiler-warnings' * origin/topic/jsiwek/compiler-warnings: Fixing compiler warnings (addresses #388) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From braun at net.in.tum.de Thu Nov 3 16:56:18 2011 From: braun at net.in.tum.de (Lothar Braun) Date: Fri, 4 Nov 2011 00:56:18 +0100 Subject: [Bro-Dev] snaplen and drops In-Reply-To: <198E3AD7-715B-461B-9EDA-6B110B20888C@icir.org> References: <20111027042458.GA87273@icir.org> <535469B6-200D-4940-A2E5-704C89457850@net.in.tum.de> <20111028033646.GA90146@icir.org> <20111028155635.GB12238@icir.org> <6655064A-DED1-47A5-B7AB-F12868211577@net.in.tum.de> <198E3AD7-715B-461B-9EDA-6B110B20888C@icir.org> Message-ID: <8F6811C9-DB0F-4520-89D8-95F8EFA7A7B1@net.in.tum.de> Hi, On Nov 3, 2011, at 2:42 PM, Seth Hall wrote: > > On Nov 3, 2011, at 8:58 AM, Lothar Braun wrote: > >> I had a quick look at the libpcap (1.2.0) and the libdaq (0.6.2) code. It seems to me that both of them perform basically the same steps for packet acquisition. >> >> Both create a socket PF_PACKET socket, both request a shared memory area on the capturing rx_ring. And both perform similar operations during packet acquisition: > > > Did you look int the os-daq-modules/daq_afpacket.c file? DAQ implements the AF_PACKET support there. Yes, I was comparing daq-0.6.2/os-daq-modules/daq_afpacket.c to libpcap-1.2.0/pcap-linux.c In my opinion, the important parts are: Setup phase: socket(PF_PACKET, SOCK_RAW, htons(ETH_P_ALL)) which creates the AF_PACKET socket setsockopt(handle->fd, SOL_PACKET, PACKET_RX_RING, some_stuff_stuff); mmap(...); which creates and maps a shared buffer between kernel and user space. One important difference could be the default buffer size between kernel and userspace. Libpcap chooses 2 MB. I have no idea what libdaq defaults to. Receiving packets is also done in a similar way: As long as there are packets: comsume packets else call poll() on the socket and sleep until new packet arrives So, if I'm not overlooking something that is important for capturing performance, both implementations should result in similar capture rates. Lothar -- Lothar Braun Chair for Network Architectures and Services (I8) Department of Informatics Technische Universit?t M?nchen Boltzmannstr. 3, 85748 Garching bei M?nchen, Germany Phone: +49 89 289-18010 Fax: +49 89 289-18033 E-mail: braun at net.in.tum.de From bro at tracker.bro-ids.org Thu Nov 3 17:00:20 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 04 Nov 2011 00:00:20 -0000 Subject: [Bro-Dev] #663: Load order local-* Message-ID: <047.b0024243ec74c2557a7442cf3f8bb03c@tracker.bro-ids.org> #663: Load order local-* ------------------------+--------------------- Reporter: robin | Type: Problem Status: new | Priority: Normal Milestone: Bro2.0 | Component: Bro Version: git/master | ------------------------+--------------------- local-manager.bro is loaded before local.bro, which quite inconvinient because it means one doesn't have access to any of the identifiers defined (@loaded) in local.bro, like for using within a notice policy. Is it possible to turn the load order around? -- Ticket URL: Bro Tracker Bro Issue Tracker From braun at net.in.tum.de Thu Nov 3 17:00:27 2011 From: braun at net.in.tum.de (Lothar Braun) Date: Fri, 4 Nov 2011 01:00:27 +0100 Subject: [Bro-Dev] snaplen and drops In-Reply-To: References: <20111027042458.GA87273@icir.org> <535469B6-200D-4940-A2E5-704C89457850@net.in.tum.de> <20111028033646.GA90146@icir.org> <20111028155635.GB12238@icir.org> <6655064A-DED1-47A5-B7AB-F12868211577@net.in.tum.de> Message-ID: <6A6CBAA7-324A-4499-A4E0-E2085F31CEEF@net.in.tum.de> Hi, On Nov 3, 2011, at 2:51 PM, Martin Holste wrote: >> So if you use libpcap >= 1.0.0, you should have AF_PACKET support by default. Snort/Suricata probably implemented separate AF_PACKET support for systems that ship libpcap < 1.0.0. > > I've used pcap > 1.0 and had much worse performance than AF_PACKET, This is interesting. Could your result be related to the small default buffer size in libpcap (the 2MB which have been problematic with Bro if a snaplen of 64 Kb is used?) Can you remember how your setup looked like? > so > I'd be willing to bet that IRQ CPU utilization is higher with pcap and > AF_PACKET does a polling mechanism to decrease its IRQ overhead. I > can't speak to the mmap techniques and whether or not they differ, but > IRQ alone would be enough to make a noticeable difference. Hmm, I'm not sure that I understand. What do you mean with IRQ? Hardware Interrupts originated from the NIC? I can see the following things that can influence the capturing performance: 1.) hardware interrupts 2.) software interrupts == availability of kernel threads to pull data into userspace 3.) packet copy operations 4.) packet exchange between kernel and userspace (e.g. mmap) 5.) synchronization between kernel and userspace (e.g. poll() on a socket) 1.) + 2.) are handled by the kernel, and to the best of my knowledge neither libpcap nor libdaq should have any influence on them. 3)-5) are done using the same mechanisms in both libraries. When I'm back at our lab in the next week, I'll try to find some time to do some experiments. If I can reproduce your observations, maybe I can find an explanation for the differences. Lothar -- Lothar Braun Chair for Network Architectures and Services (I8) Department of Informatics Technische Universit?t M?nchen Boltzmannstr. 3, 85748 Garching bei M?nchen, Germany Phone: +49 89 289-18010 Fax: +49 89 289-18033 E-mail: braun at net.in.tum.de From jsiwek at ncsa.illinois.edu Thu Nov 3 17:26:45 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Thu, 3 Nov 2011 19:26:45 -0500 (CDT) Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: No longer write to the PacketFilter::LOG stream if not reading traffic. (507b51c) In-Reply-To: <201111032233.pA3MXbiT001380@bro-ids.icir.org> Message-ID: <1026708526.1523.1320366405114.JavaMail.root@zimbra-1.ncsa.uiuc.edu> This commit is likely why the core.print-bpf-filters-ipv4 test fails now. +Jon ----- Original Message ----- > Repository : ssh://git at bro-ids.icir.org/bro > > On branches: fastpath,master,topic/robin/pp-alarms > Link : > http://tracker.bro-ids.org/bro/changeset/507b51c95750186350ddc8bfd2f3a8590e548455/bro > > >--------------------------------------------------------------- > > commit 507b51c95750186350ddc8bfd2f3a8590e548455 > Author: Seth Hall > Date: Wed Nov 2 15:09:57 2011 -0400 > > No longer write to the PacketFilter::LOG stream if not reading > traffic. > > > >--------------------------------------------------------------- > > 507b51c95750186350ddc8bfd2f3a8590e548455 > scripts/base/frameworks/packet-filter/main.bro | 3 ++- > 1 files changed, 2 insertions(+), 1 deletions(-) > > diff --git a/scripts/base/frameworks/packet-filter/main.bro > b/scripts/base/frameworks/packet-filter/main.bro > index 784a772..1097315 100644 > --- a/scripts/base/frameworks/packet-filter/main.bro > +++ b/scripts/base/frameworks/packet-filter/main.bro > @@ -144,7 +144,8 @@ function install() > $sub=default_filter]); > } > > - Log::write(PacketFilter::LOG, info); > + if ( reading_live_traffic() || reading_traces() ) > + Log::write(PacketFilter::LOG, info); > } > > event bro_init() &priority=10 > > _______________________________________________ > bro-commits mailing list > bro-commits at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-commits From robin at icir.org Thu Nov 3 17:32:13 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 3 Nov 2011 17:32:13 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: No longer write to the PacketFilter::LOG stream if not reading traffic. (507b51c) In-Reply-To: <1026708526.1523.1320366405114.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <201111032233.pA3MXbiT001380@bro-ids.icir.org> <1026708526.1523.1320366405114.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <20111104003213.GA88950@icir.org> On Thu, Nov 03, 2011 at 19:26 -0500, you wrote: > This commit is likely why the core.print-bpf-filters-ipv4 test fails now. Ah, I actually saw that, but then got distracted and forgot about the failing test. Will fix! Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From mcholste at gmail.com Fri Nov 4 05:39:04 2011 From: mcholste at gmail.com (Martin Holste) Date: Fri, 4 Nov 2011 07:39:04 -0500 Subject: [Bro-Dev] snaplen and drops In-Reply-To: <6A6CBAA7-324A-4499-A4E0-E2085F31CEEF@net.in.tum.de> References: <20111027042458.GA87273@icir.org> <535469B6-200D-4940-A2E5-704C89457850@net.in.tum.de> <20111028033646.GA90146@icir.org> <20111028155635.GB12238@icir.org> <6655064A-DED1-47A5-B7AB-F12868211577@net.in.tum.de> <6A6CBAA7-324A-4499-A4E0-E2085F31CEEF@net.in.tum.de> Message-ID: > This is interesting. Could your result be related to the small default buffer size in libpcap (the 2MB which have been problematic with Bro if a snaplen of 64 Kb is used?) > > Can you remember how your setup looked like? The setup was a stock Ubuntu 10.04 LTS on both Intel and Broadcom nics, and the behavior was observed with any libpcap-based application, including tcpdump. > Hmm, I'm not sure that I understand. What do you mean with IRQ? Hardware Interrupts originated from the NIC? > > I can see the following things that can influence the capturing performance: > > 1.) hardware interrupts > 2.) software interrupts == availability of kernel threads to pull data into userspace > 3.) packet copy operations > 4.) packet exchange between kernel and userspace (e.g. mmap) > 5.) synchronization between kernel and userspace (e.g. poll() on a socket) > > 1.) + 2.) are handled by the kernel, and to the best of my knowledge neither libpcap nor libdaq should have any influence on them. This is where PF_RING and AF_PACKET come in. They alter the way in which polling takes place at the kernel level to save hardware interrupts. > When I'm back at our lab in the next week, I'll try to find some time to do some experiments. ?If I can reproduce your observations, maybe I can find an explanation for the differences. That would be great! I'm very sure that AF_PACKET performs better than stock libpcap on Ubuntu 10.04 LTS, but I can only make these guesses as to why. From bro at tracker.bro-ids.org Sat Nov 5 12:05:26 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 05 Nov 2011 19:05:26 -0000 Subject: [Bro-Dev] #664: Interesting_Hostname_Login misleading Message-ID: <047.2320a3af2d40dde4b0878f0150e175de@tracker.bro-ids.org> #664: Interesting_Hostname_Login misleading ---------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Keywords: | ---------------------+-------------------- It generates messages of the from `interesting login from hostname X` with X the *responder* of a connection. {{{ for ( host in set(c$id$orig_h, c$id$resp_h) ) { [?] NOTICE([$note=Interesting_Hostname_Login, $msg=fmt("Interesting login from hostname: %s", hostname), $sub=hostname, $conn=c]); } } }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Nov 5 18:54:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 06 Nov 2011 01:54:49 -0000 Subject: [Bro-Dev] #664: Interesting_Hostname_Login misleading In-Reply-To: <047.2320a3af2d40dde4b0878f0150e175de@tracker.bro-ids.org> References: <047.2320a3af2d40dde4b0878f0150e175de@tracker.bro-ids.org> Message-ID: <062.9b3ec063537fd9aee3e38557fafc6480@tracker.bro-ids.org> #664: Interesting_Hostname_Login misleading ----------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by seth): > It generates messages of the from `interesting login from hostname X` with > X the *responder* of a connection. Thanks for filing the ticket. I've been meaning to fix that for a while. Does anyone have better wording for the message? How about "SSH login involving a host with an interesting name." -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Nov 6 19:18:19 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 07 Nov 2011 03:18:19 -0000 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> Message-ID: <062.c70795bc9fa7a63a4a8359598915331d@tracker.bro-ids.org> #388: Fix more compiler warnings -----------------------------+-------------------- Reporter: robin | Owner: jsiwek Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by robin): * status: assigned => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Nov 6 19:27:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 07 Nov 2011 03:27:05 -0000 Subject: [Bro-Dev] #613: Problem with LogExpireInterval setting in BroControl In-Reply-To: <046.6b2381d7e3479123e85b6eccb468740e@tracker.bro-ids.org> References: <046.6b2381d7e3479123e85b6eccb468740e@tracker.bro-ids.org> Message-ID: <061.5a923187484709461a6c558da70f7210@tracker.bro-ids.org> #613: Problem with LogExpireInterval setting in BroControl -------------------------+-------------------- Reporter: seth | Owner: robin Type: Problem | Status: closed Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: Resolution: fixed | Keywords: -------------------------+-------------------- Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [28ababa5597de95fb4830638ab9cc187d8523714/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="28ababa5597de95fb4830638ab9cc187d8523714" Merge remote-tracking branch 'origin/fastpath' * origin/fastpath: broctl.cfg now determines sendmail location at configure-time Disable log expiration by default. (Addresses #613) Make symlink to broctl-config.sh update with `broctl install` Closes #613. Closes #648. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Nov 6 19:27:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 07 Nov 2011 03:27:05 -0000 Subject: [Bro-Dev] #648: SpoolDir setting isn't used everywhere In-Reply-To: <046.ea9ac6eac3f6ed983568e802720fbd53@tracker.bro-ids.org> References: <046.ea9ac6eac3f6ed983568e802720fbd53@tracker.bro-ids.org> Message-ID: <061.1349c7c8e340fe6980909579de8df82a@tracker.bro-ids.org> #648: SpoolDir setting isn't used everywhere -------------------------+-------------------- Reporter: seth | Owner: robin Type: Problem | Status: closed Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: Resolution: fixed | Keywords: -------------------------+-------------------- Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [28ababa5597de95fb4830638ab9cc187d8523714/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="28ababa5597de95fb4830638ab9cc187d8523714" Merge remote-tracking branch 'origin/fastpath' * origin/fastpath: broctl.cfg now determines sendmail location at configure-time Disable log expiration by default. (Addresses #613) Make symlink to broctl-config.sh update with `broctl install` Closes #613. Closes #648. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Nov 7 02:29:46 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 07 Nov 2011 10:29:46 -0000 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> Message-ID: <062.9d8d6d18e40e2be66bd289568c9e1399@tracker.bro-ids.org> #388: Fix more compiler warnings ----------------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Merge Request | Status: reopened Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------------+---------------------- Changes (by jsiwek): * status: closed => reopened * resolution: Solved/Applied => Comment: Did you look at the `topic/jsiwek/compiler-warnings` branches in the `broccoli-ruby`, `broccoli`, and `bro-aux` repos? It looks like those may not have been merged and there's still some warnings emitted from them on certain platforms. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Nov 7 05:40:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 07 Nov 2011 13:40:50 -0000 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> Message-ID: <062.aeb16b61ea0b1c1866ca3b065ece139f@tracker.bro-ids.org> #388: Fix more compiler warnings ----------------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Merge Request | Status: reopened Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------------+---------------------- Comment (by robin): On Mon, Nov 07, 2011 at 10:29 -0000, you wrote: > Did you look at the `topic/jsiwek/compiler-warnings` branches in the > `broccoli-ruby`, `broccoli`, and `bro-aux` repos? It looks like those may > not have been merged and there's still some warnings emitted from them on > certain platforms. I did, looks like something went wrong with my merges if they aren't there. Will check. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Nov 7 05:57:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 07 Nov 2011 13:57:34 -0000 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> Message-ID: <062.958838342200157fa942d4eabb299192@tracker.bro-ids.org> #388: Fix more compiler warnings ----------------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Merge Request | Status: reopened Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------------+---------------------- Comment (by robin): In [906f970df5f708582c7002069b787d5af586b46f/bro-aux]: {{{ #!CommitTicketReference repository="bro-aux" revision="906f970df5f708582c7002069b787d5af586b46f" Merge remote-tracking branch 'origin/topic/jsiwek/compiler-warnings' * origin/topic/jsiwek/compiler-warnings: Fixing compiler warnings (addresses #388) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Nov 7 05:57:42 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 07 Nov 2011 13:57:42 -0000 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> Message-ID: <062.dff988f906baca1ca2c4cdea67af16d6@tracker.bro-ids.org> #388: Fix more compiler warnings ----------------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Merge Request | Status: reopened Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------------+---------------------- Comment (by robin): In [4a1fc7c181c92be6878d42cdaa7465f64c684fdd/broccoli]: {{{ #!CommitTicketReference repository="broccoli" revision="4a1fc7c181c92be6878d42cdaa7465f64c684fdd" Merge remote-tracking branch 'origin/topic/jsiwek/compiler-warnings' * origin/topic/jsiwek/compiler-warnings: Fixing compiler warnings (addresses #388) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Nov 7 05:57:55 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 07 Nov 2011 13:57:55 -0000 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> Message-ID: <062.ebf217ecb35ab334f2dc9c9a24bb040b@tracker.bro-ids.org> #388: Fix more compiler warnings ----------------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Merge Request | Status: reopened Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------------+---------------------- Comment (by robin): In [3091c493d4691d72dc4d890ba7d9ae728db87e26/broccoli-ruby]: {{{ #!CommitTicketReference repository="broccoli-ruby" revision="3091c493d4691d72dc4d890ba7d9ae728db87e26" Merge remote-tracking branch 'origin/topic/jsiwek/compiler-warnings' * origin/topic/jsiwek/compiler-warnings: Ignoring some SWIG warnings (addresses #388) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Nov 7 06:35:08 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 07 Nov 2011 14:35:08 -0000 Subject: [Bro-Dev] #665: broctl and check/install/(re)start dance refinement Message-ID: <046.d4c1207e818d02e104cbc6fa4ddc5247@tracker.bro-ids.org> #665: broctl and check/install/(re)start dance refinement ------------------------+----------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: BroControl | Version: Keywords: | ------------------------+----------------- It would make broctl more easily documented and understood if broctl automatically ran the install command at start up if the broctl.dat file doesn't exist. Right now there is a special case for check/install/restart since if it's the first time someone has run broctl, they need to do install/check/start. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Nov 8 08:46:18 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 08 Nov 2011 16:46:18 -0000 Subject: [Bro-Dev] #666: broctl + macports python libraries Message-ID: <048.d70212220b0f1740e974880807144f64@tracker.bro-ids.org> #666: broctl + macports python libraries ------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: git/master Keywords: | ------------------------+------------------------ I've seen a few errors that look like broctl gets configured/installed with a hashbang that uses the standard python that ships with OS X, while at build time the pybroccoli bindings get built against the macports version of python libraries. The result is an abort trap at runtime since`broctl` uses a mismatch of python interpreter and python-libraries as linked by the broccoli bindings. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Nov 8 13:42:30 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 08 Nov 2011 21:42:30 -0000 Subject: [Bro-Dev] #667: Non-deterministic behavior when deleting current set element during iteration Message-ID: <054.b8c40b9042767a95b701e6894c97ee6e@tracker.bro-ids.org> #667: Non-deterministic behavior when deleting current set element during iteration --------------------------+--------------------- Reporter: david.bianco | Type: Problem Status: new | Priority: Normal Milestone: | Component: Bro Version: 2.0 Beta | Keywords: --------------------------+--------------------- As we discussed during the Bro Workshop at NCSA, the attached code shows some non-deterministic results while deleting the current element while iterating through a set. Most of the time, it works. Some times, it doesn't. Have a look at subexercise 3's output. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Nov 8 14:16:26 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 08 Nov 2011 22:16:26 -0000 Subject: [Bro-Dev] #667: Non-deterministic behavior when deleting current set element during iteration In-Reply-To: <054.b8c40b9042767a95b701e6894c97ee6e@tracker.bro-ids.org> References: <054.b8c40b9042767a95b701e6894c97ee6e@tracker.bro-ids.org> Message-ID: <069.d387eab65d52c4c9e4bb81ef74f2ba0e@tracker.bro-ids.org> #667: Non-deterministic behavior when deleting current set element during iteration ---------------------------+---------------------- Reporter: david.bianco | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: 2.0 Beta Resolution: | Keywords: ---------------------------+---------------------- Comment (by vallenti): That's an interesting bug, which probably boils down to the following issue: {{{ global s: set[T]; for (i in s) { delete s[i]; # ... } }}} Here, we delete the very element of the current iteration in the for loop. This probably invalidates C++ iterators and is causing the non- deterministic behavior. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Nov 8 16:41:29 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 09 Nov 2011 00:41:29 -0000 Subject: [Bro-Dev] #662: Patch to make BroControl Python 2.4 friendly In-Reply-To: <047.74c9f68692c0cdac77f6fb2c362eb07c@tracker.bro-ids.org> References: <047.74c9f68692c0cdac77f6fb2c362eb07c@tracker.bro-ids.org> Message-ID: <062.7a584cf8528dc75ef0a9610a36743443@tracker.bro-ids.org> #662: Patch to make BroControl Python 2.4 friendly -------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Patch | Status: closed Priority: Normal | Milestone: 2.0 Component: BroControl | Version: git/master Resolution: fixed | Keywords: -------------------------+------------------------ Changes (by jsiwek): * owner: => jsiwek * status: new => closed * resolution: => fixed Comment: In [64933d30931652276fa346d6ae19e8580fc9135b/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="64933d30931652276fa346d6ae19e8580fc9135b" Apply patch for BroControl Python 2.3/2.4 compat by William Jones. Closes #662 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 9 04:46:42 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 09 Nov 2011 12:46:42 -0000 Subject: [Bro-Dev] #666: broctl + macports python libraries In-Reply-To: <048.d70212220b0f1740e974880807144f64@tracker.bro-ids.org> References: <048.d70212220b0f1740e974880807144f64@tracker.bro-ids.org> Message-ID: <063.58e145bc2084997f4ca33fd73700b4f7@tracker.bro-ids.org> #666: broctl + macports python libraries -------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: git/master Resolution: | Keywords: -------------------------+------------------------ Comment (by jsiwek): In [142043b65f447ac0965aee78158c2953b6f01017/pysubnettree]: {{{ #!CommitTicketReference repository="pysubnettree" revision="142043b65f447ac0965aee78158c2953b6f01017" Simplify finding of Python headers/libraries. They'll also more closely follow the python interpreter that's found now as the FindPythonDev script knows to make use of python-config Addresses #666 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 9 04:48:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 09 Nov 2011 12:48:24 -0000 Subject: [Bro-Dev] #666: broctl + macports python libraries In-Reply-To: <048.d70212220b0f1740e974880807144f64@tracker.bro-ids.org> References: <048.d70212220b0f1740e974880807144f64@tracker.bro-ids.org> Message-ID: <063.6971de434dba4b06ef4df77291889eac@tracker.bro-ids.org> #666: broctl + macports python libraries -------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: git/master Resolution: | Keywords: -------------------------+------------------------ Comment (by jsiwek): In [6737c69dafa90fd969b15f0c954098a5e8db17ad/broccoli-python]: {{{ #!CommitTicketReference repository="broccoli-python" revision="6737c69dafa90fd969b15f0c954098a5e8db17ad" Simplify finding of Python headers/libraries. They'll also more closely follow the python interpreter that's found now as the FindPythonDev script knows to make use of python-config Addresses #666 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 9 05:04:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 09 Nov 2011 13:04:13 -0000 Subject: [Bro-Dev] #666: broctl + macports python libraries In-Reply-To: <048.d70212220b0f1740e974880807144f64@tracker.bro-ids.org> References: <048.d70212220b0f1740e974880807144f64@tracker.bro-ids.org> Message-ID: <063.9b825c0e92b3f1e7f740b946a258b7df@tracker.bro-ids.org> #666: broctl + macports python libraries ----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Merge Request | Status: new Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * type: Problem => Merge Request Comment: The `topic/jsiwek/find-pythondev` branch in repos `cmake`, `broccoli- python`, and `pysubnettree` generally simplifies the process of finding Python libraries through use of python-config (if not available, it falls back on old FindPythonLibs call). And since python-config can also be derived from the location of the python interpreter, this makes it harder to mismatch libraries and interpreter versions. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 9 06:55:52 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 09 Nov 2011 14:55:52 -0000 Subject: [Bro-Dev] #668: Please support UTC conversion in bro-cut Message-ID: <054.642e0955ca3b58a1649cb85169342f62@tracker.bro-ids.org> #668: Please support UTC conversion in bro-cut --------------------------+----------------------------- Reporter: david.bianco | Type: Feature Request Status: new | Priority: Normal Milestone: | Component: Bro Version: 2.0 Beta | Keywords: --------------------------+----------------------------- For those of us who deal with sensors in different timezones, it would be convenient to have bro-cut (or its successor utility) support automatic conversion of timestamps to UTC instead of only the local time. This would help a lot when creating timelines of events that happen in multiple places. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 9 06:59:31 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 09 Nov 2011 14:59:31 -0000 Subject: [Bro-Dev] #669: Bro-cut fails when no gawk is present Message-ID: <054.8d38b264fc49b0372d0d839413ccf3c4@tracker.bro-ids.org> #669: Bro-cut fails when no gawk is present --------------------------+----------------------------- Reporter: david.bianco | Type: Feature Request Status: new | Priority: Normal Milestone: | Component: Bro Version: 2.0 Beta | Keywords: --------------------------+----------------------------- I know bro-cut depends on gawk, but if gawk is not present on the system, it is not obvious from the bro-cut errors. Please have bro-cut check for the presence of gawk and emit an easy-to-understand message that gawk is required. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 9 07:06:56 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 09 Nov 2011 15:06:56 -0000 Subject: [Bro-Dev] #668: Please support UTC conversion in bro-cut In-Reply-To: <054.642e0955ca3b58a1649cb85169342f62@tracker.bro-ids.org> References: <054.642e0955ca3b58a1649cb85169342f62@tracker.bro-ids.org> Message-ID: <069.9b797c2ea2812dbd3e5af0469cff2c2c@tracker.bro-ids.org> #668: Please support UTC conversion in bro-cut ------------------------------+---------------------- Reporter: david.bianco | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: 2.0 Beta Resolution: | Keywords: ------------------------------+---------------------- Changes (by robin): * milestone: => Bro2.0 -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 9 07:13:03 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 09 Nov 2011 15:13:03 -0000 Subject: [Bro-Dev] #669: Bro-cut fails when no gawk is present In-Reply-To: <054.8d38b264fc49b0372d0d839413ccf3c4@tracker.bro-ids.org> References: <054.8d38b264fc49b0372d0d839413ccf3c4@tracker.bro-ids.org> Message-ID: <069.6b8d0f612c5f9d2d584f51beeb8cd8b6@tracker.bro-ids.org> #669: Bro-cut fails when no gawk is present ------------------------------+---------------------- Reporter: david.bianco | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Component: Bro | Version: 2.0 Beta Resolution: | Keywords: ------------------------------+---------------------- Comment (by robin): > I know bro-cut depends on gawk, but if gawk is not present on the system, > it is not obvious from the bro-cut errors. Please have bro-cut check for > the presence of gawk and emit an easy-to-understand message that gawk is > required. Actually it shouldn't depend on gawk. It needs gawk just for doing the time conversion, but can fall back to standard awk if that's not needed. If -d is used but no gawk found, there should be a reasonable error message. So this sounds like a bug. Can you add the error message you're seeing to the ticket, and also the output of "bash -x bro-cut "? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 9 07:15:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 09 Nov 2011 15:15:28 -0000 Subject: [Bro-Dev] #669: Bro-cut fails when no gawk is present In-Reply-To: <054.8d38b264fc49b0372d0d839413ccf3c4@tracker.bro-ids.org> References: <054.8d38b264fc49b0372d0d839413ccf3c4@tracker.bro-ids.org> Message-ID: <069.4dd917a215120ec3358020135972fc94@tracker.bro-ids.org> #669: Bro-cut fails when no gawk is present ---------------------------+---------------------- Reporter: david.bianco | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: 2.0 Beta Resolution: | Keywords: ---------------------------+---------------------- Changes (by robin): * type: Feature Request => Problem * milestone: => Bro2.0 -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 9 08:41:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 09 Nov 2011 16:41:13 -0000 Subject: [Bro-Dev] #669: Bro-cut fails when no gawk is present In-Reply-To: <054.8d38b264fc49b0372d0d839413ccf3c4@tracker.bro-ids.org> References: <054.8d38b264fc49b0372d0d839413ccf3c4@tracker.bro-ids.org> Message-ID: <069.82038f078469f8a955a0a6db1323c3ee@tracker.bro-ids.org> #669: Bro-cut fails when no gawk is present ---------------------------+---------------------- Reporter: david.bianco | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: 2.0 Beta Resolution: | Keywords: ---------------------------+---------------------- Comment (by dnthayer): This is a duplicate of ticket 653. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 9 14:06:33 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 09 Nov 2011 22:06:33 -0000 Subject: [Bro-Dev] #670: Extend decode_base64() to accomodate alternate base64 charsets Message-ID: <054.5c962cbdb24e1dffa88a8640cda68c12@tracker.bro-ids.org> #670: Extend decode_base64() to accomodate alternate base64 charsets --------------------------+----------------------------- Reporter: david.bianco | Type: Feature Request Status: new | Priority: Normal Milestone: | Component: Bro Version: 2.0 Beta | Keywords: --------------------------+----------------------------- Using an alternate base64 alphabet to encode data with the standard base64 algorithm is effectively the same as encrypting it with a long alphanumeric encryption key. We see this in common use in malware C2 channels. If we could supply a known alphabet to the base64 function, we could more easily decode some of the C2 channels. This would be of great benefit to many Bro users. Ideally, I'd like to see something like: const my_alphabet: string = "..."; decoded = decode_base64(data, my_alphabet); -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 9 16:07:59 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 10 Nov 2011 00:07:59 -0000 Subject: [Bro-Dev] #671: Test Bro core and script layer simultaneously Message-ID: <050.19b531854c8659a808d1449a532c1e42@tracker.bro-ids.org> #671: Test Bro core and script layer simultaneously ----------------------+------------------------ Reporter: vallenti | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.1 Component: BTest | Version: git/master Keywords: | ----------------------+------------------------ If we record all events during testing, say by adding `events.bst` to each Bro run, we can simultaneously test the core. Moreover, we instantly know whether a bug manifests at script land or at the core. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Nov 10 06:00:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 10 Nov 2011 14:00:05 -0000 Subject: [Bro-Dev] #670: Extend decode_base64() to accomodate alternate base64 charsets In-Reply-To: <054.5c962cbdb24e1dffa88a8640cda68c12@tracker.bro-ids.org> References: <054.5c962cbdb24e1dffa88a8640cda68c12@tracker.bro-ids.org> Message-ID: <069.d015eb739df87b2cb856845fe0ce2f3e@tracker.bro-ids.org> #670: Extend decode_base64() to accomodate alternate base64 charsets ------------------------------+---------------------- Reporter: david.bianco | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: 2.0 Beta Resolution: | Keywords: ------------------------------+---------------------- Changes (by robin): * milestone: => Bro2.0 Comment: Should be rather easy to do. Let's see if we can get that into 2.0 final. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Nov 10 12:40:57 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 10 Nov 2011 20:40:57 -0000 Subject: [Bro-Dev] #672: Bring POP3 back into the distribution Message-ID: <050.279fcf58665a40f49f7d04afea1d1ce3@tracker.bro-ids.org> #672: Bring POP3 back into the distribution ----------------------+------------------------ Reporter: vallenti | Owner: seth Type: Task | Status: new Priority: Normal | Milestone: Bro2.1 Component: Bro | Version: git/master Keywords: | ----------------------+------------------------ The current master has no longer support for POP3. It lingers around but we need to bring it back into the distribution. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Nov 11 08:02:06 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 11 Nov 2011 16:02:06 -0000 Subject: [Bro-Dev] #670: Extend decode_base64() to accomodate alternate base64 charsets In-Reply-To: <054.5c962cbdb24e1dffa88a8640cda68c12@tracker.bro-ids.org> References: <054.5c962cbdb24e1dffa88a8640cda68c12@tracker.bro-ids.org> Message-ID: <069.493f1f1d8681d45322d13ce2529df7de@tracker.bro-ids.org> #670: Extend decode_base64() to accomodate alternate base64 charsets ------------------------------+---------------------- Reporter: david.bianco | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: 2.0 Beta Resolution: | Keywords: ------------------------------+---------------------- Comment (by jsiwek): I just wrote some code to do this. Anyone have a quick example of an alternate alphabet with sample encoded/decoded strings so I can write a test for it? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Nov 11 09:34:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 11 Nov 2011 17:34:50 -0000 Subject: [Bro-Dev] #673: topic/jsiwek/parallel-make-recursion Message-ID: <048.5084a3ff12f6165e7c14d33fcb4d9d11@tracker.bro-ids.org> #673: topic/jsiwek/parallel-make-recursion ---------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Low | Milestone: Bro2.0 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ This branch is in repos `bro`, `binpac`, `broccoli`, `bro-aux`, `broctl`, and `capstats` and seems to fix the top-level `Makefile` problems with parallel builds (the `-j` option) being ignored on some platforms. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Nov 11 10:42:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 11 Nov 2011 18:42:05 -0000 Subject: [Bro-Dev] #674: topic/jsiwek/require-libmagic-libz Message-ID: <048.10e00f6a9511cdabaf15c7045c6e7a2d@tracker.bro-ids.org> #674: topic/jsiwek/require-libmagic-libz ---------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Low | Milestone: Bro2.0 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ This branch promotes libmagic and libz to be required instead of optional dependencies. Reason being that things work poorly without them. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Nov 11 11:04:40 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 11 Nov 2011 19:04:40 -0000 Subject: [Bro-Dev] #645: Figure out where sendmail is In-Reply-To: <047.30bfd70cef14df8f513f2d5595dceaae@tracker.bro-ids.org> References: <047.30bfd70cef14df8f513f2d5595dceaae@tracker.bro-ids.org> Message-ID: <062.c0503660f3e70f392ca5df11a41572bd@tracker.bro-ids.org> #645: Figure out where sendmail is -------------------------+-------------------- Reporter: robin | Owner: jsiwek Type: Problem | Status: closed Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: Resolution: fixed | Keywords: -------------------------+-------------------- Changes (by jsiwek): * status: assigned => closed * resolution: => fixed Comment: In [534d908735699e0fa6572a6b059c3a5c78470df7/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="534d908735699e0fa6572a6b059c3a5c78470df7" Configure sendmail option in options.py instead of broctl.cfg (fixes #645) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Nov 11 11:50:10 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 11 Nov 2011 19:50:10 -0000 Subject: [Bro-Dev] #670: Extend decode_base64() to accomodate alternate base64 charsets In-Reply-To: <054.5c962cbdb24e1dffa88a8640cda68c12@tracker.bro-ids.org> References: <054.5c962cbdb24e1dffa88a8640cda68c12@tracker.bro-ids.org> Message-ID: <069.7b88a264a007a9bc1fa0b8a9aa40ba94@tracker.bro-ids.org> #670: Extend decode_base64() to accomodate alternate base64 charsets ------------------------------+---------------------- Reporter: david.bianco | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: 2.0 Beta Resolution: | Keywords: ------------------------------+---------------------- Comment (by jsiwek): In [5865bf38501fe892c29302916e022dbb855c94fa/bro]: {{{ #!CommitTicketReference repository="bro" revision="5865bf38501fe892c29302916e022dbb855c94fa" Add decode_base64_custom BiF to allow alternate base64 alphabets. Addresses #670 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Nov 11 11:57:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 11 Nov 2011 19:57:05 -0000 Subject: [Bro-Dev] #670: Extend decode_base64() to accomodate alternate base64 charsets In-Reply-To: <054.5c962cbdb24e1dffa88a8640cda68c12@tracker.bro-ids.org> References: <054.5c962cbdb24e1dffa88a8640cda68c12@tracker.bro-ids.org> Message-ID: <069.705ffd2ef995d488f668fd86e0d11133@tracker.bro-ids.org> #670: Extend decode_base64() to accomodate alternate base64 charsets ----------------------------+---------------------- Reporter: david.bianco | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: 2.0 Beta Resolution: | Keywords: ----------------------------+---------------------- Changes (by jsiwek): * type: Feature Request => Merge Request Comment: > Anyone have a quick example of an alternate alphabet with sample encoded/decoded strings so I can write a test for it? Nevermind, I manually made something that checked out ok, but if someone has a real example, it could be added to the new test. The changes are in `topic/jsiwek/custom-b64-alphabet` -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Nov 14 17:20:41 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 15 Nov 2011 01:20:41 -0000 Subject: [Bro-Dev] #675: enum values in log output can be ambitious Message-ID: <047.4f1dd8bfa80747c53415ac796804b4c9@tracker.bro-ids.org> #675: enum values in log output can be ambitious ---------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Keywords: | ---------------------+-------------------- Assume code like this extending enums: {{{ module M1; redef enum Other::MyEnum += { green }; module M2; redef enum Other::MyEnum += { green }; }}} That generates two separate enum labels internally, but the in the log output both appear as `green`. Normally it probably won't matter much, but now reading it back via the input framework is ambitious, Not sure what the best solution is. I'm thinking to leave everything as it is except when we actually do have such ambiguous labels: in that case, we could write out the fully scoped `M1::green`. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Nov 15 08:08:31 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 15 Nov 2011 16:08:31 -0000 Subject: [Bro-Dev] #645: Figure out where sendmail is In-Reply-To: <047.30bfd70cef14df8f513f2d5595dceaae@tracker.bro-ids.org> References: <047.30bfd70cef14df8f513f2d5595dceaae@tracker.bro-ids.org> Message-ID: <062.0541584a13a9e3ba5830eb8e1365ccf8@tracker.bro-ids.org> #645: Figure out where sendmail is -------------------------+-------------------- Reporter: robin | Owner: jsiwek Type: Problem | Status: closed Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: Resolution: fixed | Keywords: -------------------------+-------------------- Comment (by robin): In [6fb4e5689d2ae0d1c4ab7af0a8df80e6eaa98fb6/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="6fb4e5689d2ae0d1c4ab7af0a8df80e6eaa98fb6" Merge remote-tracking branch 'origin/fastpath' * origin/fastpath: Add "C CXX" to CMake project(). Configure sendmail option in options.py instead of broctl.cfg (fixes #645) Fix extraneous installation of BroControl plugins. Apply patch for BroControl Python 2.3/2.4 compat by William Jones. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Nov 15 09:14:55 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 15 Nov 2011 17:14:55 -0000 Subject: [Bro-Dev] #676: Broctl not relinking logs/current correctly Message-ID: <046.cd0ffbb5c01e8b46c16ffc30a036451e@tracker.bro-ids.org> #676: Broctl not relinking logs/current correctly ------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: Keywords: | ------------------------+-------------------- If you change from a standalone instance to a cluster, the logs/current directory will be left at the standalone path (spool/bro) and not updated to the new and correct path (spool/manager). -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Nov 15 11:00:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 15 Nov 2011 19:00:13 -0000 Subject: [Bro-Dev] #677: diff-all problem with testing Message-ID: <047.9158b91f8c87419388ef4a41186b3862@tracker.bro-ids.org> #677: diff-all problem with testing ---------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Keywords: | ---------------------+-------------------- Running `testing/external/scripts/diff-all *.log` in the `external/` tests doesn't catch if Bro doesn't produce an output file that the Baseline expects to be around. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Nov 15 12:13:46 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 15 Nov 2011 20:13:46 -0000 Subject: [Bro-Dev] #678: Fix and test Bro's debugger Message-ID: <046.da1d3fc255d7c02621c3a842f4054072@tracker.bro-ids.org> #678: Fix and test Bro's debugger ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.2 Component: Bro | Version: Keywords: | ---------------------+-------------------- Vern commented a while back that the debugger is currently broken in Bro. Let's get it working again and tested. If someone feels like fixing this up for the 2.1 release we can certainly bump it forward but I'm going to target it at 2.2 for now. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Nov 15 12:40:30 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 15 Nov 2011 20:40:30 -0000 Subject: [Bro-Dev] #679: Add support for deleting vector elements Message-ID: <050.0e994e46e1e8e7343905449869e3f2a1@tracker.bro-ids.org> #679: Add support for deleting vector elements ----------------------+------------------------ Reporter: matthias | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: git/master Keywords: | ----------------------+------------------------ The following code should work but doesn't: {{{ global v: vector of string; bro_init() { v[3] = "foo"; delete v[3]; } }}} It gives the following error message: {{{ error in ./vector.bro, line 6: illegal delete statement (delete v[3]) }}} Sets and tables support the `delete` operation, and an intuitive brogrammer expects the same thing to work with vectors as well. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Nov 15 14:36:47 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 15 Nov 2011 22:36:47 -0000 Subject: [Bro-Dev] #650: Finish Broxygen docs In-Reply-To: <047.9abf8a0851b3b85991457973560c772f@tracker.bro-ids.org> References: <047.9abf8a0851b3b85991457973560c772f@tracker.bro-ids.org> Message-ID: <062.d933d109bcce2c0062a8838dfb22be9f@tracker.bro-ids.org> #650: Finish Broxygen docs ----------------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------------+---------------------- Changes (by jsiwek): * type: Task => Merge Request Comment: The `topic/jsiwek/broxygen-cleanup` branch gets pretty far into polishing/finishing the broxygen docs and there's enough done merging it will probably be useful to other people working on documentation and also useful for me to get feedback on other specific things that need to be done (e.g. any style issues or more documents from `www` that can be integrated into broxygen?). This doesn't address how to integrate the new broxygen docs into `www` at all yet. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Nov 15 16:48:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 16 Nov 2011 00:48:36 -0000 Subject: [Bro-Dev] #388: Fix more compiler warnings In-Reply-To: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> References: <047.179fc98dc41c3ce74c9f5d347bf34b3d@tracker.bro-ids.org> Message-ID: <062.b7dd28db328341dfffc8a795519b5958@tracker.bro-ids.org> #388: Fix more compiler warnings -----------------------------+-------------------- Reporter: robin | Owner: jsiwek Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by robin): * status: reopened => closed * resolution: => Solved/Applied Comment: Already merged, but wasn't closed yet. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Nov 15 18:45:03 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 16 Nov 2011 02:45:03 -0000 Subject: [Bro-Dev] #680: Add session_ticket handshake type to Bro's SSL/TLS parser Message-ID: <046.4d83677f8aacaea128fb59b77f77bb9d@tracker.bro-ids.org> #680: Add session_ticket handshake type to Bro's SSL/TLS parser -----------------------------+-------------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro2.1 Component: Bro | Version: Keywords: | -----------------------------+-------------------- From: http://www.ietf.org/rfc/rfc5077 The handshake is used by the server to send the session_ticket opaque data structure to clients. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 16 08:31:19 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 16 Nov 2011 16:31:19 -0000 Subject: [Bro-Dev] #681: Control characters is ASCII output Message-ID: <047.094925a06443edab94de1f00a1c21805@tracker.bro-ids.org> #681: Control characters is ASCII output ---------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Keywords: | ---------------------+-------------------- Im still seeing occasional `\x09` in the logging output. It's not reproducible but happens from time to time with different files produced by the `external` tests. I valgrined and it might be triggered by what's reported here: {{{ ==3494== Invalid read of size 1 ==3494== at 0x52C865: ODesc::AddBytes(void const*, unsigned int) (Desc.cc:264) ==3494== by 0x5AA134: LogWriterAscii::DoWriteOne(ODesc*, LogVal*, LogField const*) (Desc.h:63) ==3494== by 0x5AA449: LogWriterAscii::DoWrite(int, LogField const* const*, LogVal**) (LogWriterAscii.cc:272) ==3494== by 0x5A96D2: LogWriter::Write(int, LogVal**) (LogWriter.cc:65) ==3494== by 0x5A77A7: LogMgr::Write(EnumVal*, RecordVal*) (LogMgr.cc:1041) ==3494== by 0x56C605: BifFunc::Log::bro___write(Frame*, ValPList*) (logging.bif:46) ==3494== by 0x56913E: BuiltinFunc::Call(ValPList*, Frame*) const (Func.cc:476) ==3494== by 0x560515: CallExpr::Eval(Frame*) const (Expr.cc:4651) ==3494== by 0x5FD938: StmtList::Exec(Frame*, stmt_flow_type&) const (Stmt.cc:1402) ==3494== by 0x582C75: BroFunc::Call(ValPList*, Frame*) const (Func.cc:333) ==3494== by 0x560515: CallExpr::Eval(Frame*) const (Expr.cc:4651) ==3494== by 0x600F7F: ExprStmt::Exec(Frame*, stmt_flow_type&) const (Stmt.cc:367) ==3494== Address 0x79e6751 is 0 bytes after a block of size 1 alloc'd ==3494== at 0x4A059F7: operator new[](unsigned long) (vg_replace_malloc.c:305) ==3494== by 0x5A9F50: LogWriterAscii::LogWriterAscii() (LogWriterAscii.cc:53) ==3494== by 0x5A8269: LogWriterAscii::Instantiate() (in /da/home/robin/bro/master/build/src/bro) ==3494== by 0x5A69C6: LogMgr::CreateWriter(EnumVal*, EnumVal*, std::string, int, LogField**) (LogMgr.cc:1278) ==3494== by 0x5A7B22: LogMgr::Write(EnumVal*, RecordVal*) (LogMgr.cc:1001) ==3494== by 0x56C605: BifFunc::Log::bro___write(Frame*, ValPList*) (logging.bif:46) ==3494== by 0x56913E: BuiltinFunc::Call(ValPList*, Frame*) const (Func.cc:476) ==3494== by 0x560515: CallExpr::Eval(Frame*) const (Expr.cc:4651) ==3494== by 0x5FD938: StmtList::Exec(Frame*, stmt_flow_type&) const (Stmt.cc:1402) ==3494== by 0x582C75: BroFunc::Call(ValPList*, Frame*) const (Func.cc:333) ==3494== by 0x560515: CallExpr::Eval(Frame*) const (Expr.cc:4651) ==3494== by 0x600F7F: ExprStmt::Exec(Frame*, stmt_flow_type&) const (Stmt.cc:367) }}} there are some more of these, but they all look similar. This was generated running with connection `MGyaVigCcXk` from the `bro- testing`. (We discussed this before but I don't think a ticket was filed; at least I don't find one). -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 16 08:40:32 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 16 Nov 2011 16:40:32 -0000 Subject: [Bro-Dev] #650: Finish Broxygen docs In-Reply-To: <047.9abf8a0851b3b85991457973560c772f@tracker.bro-ids.org> References: <047.9abf8a0851b3b85991457973560c772f@tracker.bro-ids.org> Message-ID: <062.1bd2c85d5dd90c848feeb5a7763572b4@tracker.bro-ids.org> #650: Finish Broxygen docs ----------------------------+-------------------- Reporter: robin | Owner: jsiwek Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: fixed | Keywords: ----------------------------+-------------------- Changes (by robin): * status: assigned => closed * resolution: => fixed Comment: In [6b207c9c956f6fee69e8214b848610e450523e18/bro]: {{{ #!CommitTicketReference repository="bro" revision="6b207c9c956f6fee69e8214b848610e450523e18" Merge remote-tracking branch 'origin/topic/jsiwek/broxygen-cleanup' * origin/topic/jsiwek/broxygen-cleanup: Fix documentation interlinkage and normalize console directive bodies. Adapt more www niceties to broxygen (stylesheets, pygments); add more docs Finished dissolving the sphinx source directory into doc/ and doc/scripts/ Move sphinx source tree up a level. Broxygen doc style tweaks. Better adaptation of Bro website style in Broxygen docs. Start Broxygen cleanup with crude adaptation of website styles. Closes #650. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 16 08:40:33 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 16 Nov 2011 16:40:33 -0000 Subject: [Bro-Dev] #673: topic/jsiwek/parallel-make-recursion In-Reply-To: <048.5084a3ff12f6165e7c14d33fcb4d9d11@tracker.bro-ids.org> References: <048.5084a3ff12f6165e7c14d33fcb4d9d11@tracker.bro-ids.org> Message-ID: <063.51002b283b2a6092b4f6485cd19d5ba7@tracker.bro-ids.org> #673: topic/jsiwek/parallel-make-recursion ----------------------------+------------------------ Reporter: jsiwek | Owner: robin Type: Merge Request | Status: closed Priority: Low | Milestone: Bro2.0 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [6b70f340aa98f4ce8699e8d4d0402b06bb4974a1/bro]: {{{ #!CommitTicketReference repository="bro" revision="6b70f340aa98f4ce8699e8d4d0402b06bb4974a1" Merge remote-tracking branch 'origin/topic/jsiwek/parallel-make-recursion' * origin/topic/jsiwek/parallel-make-recursion: Fix parallel make from top-level to work on more platforms Closes #673. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 16 08:40:32 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 16 Nov 2011 16:40:32 -0000 Subject: [Bro-Dev] #670: Extend decode_base64() to accomodate alternate base64 charsets In-Reply-To: <054.5c962cbdb24e1dffa88a8640cda68c12@tracker.bro-ids.org> References: <054.5c962cbdb24e1dffa88a8640cda68c12@tracker.bro-ids.org> Message-ID: <069.738683b20c5ec656c456859ec8656992@tracker.bro-ids.org> #670: Extend decode_base64() to accomodate alternate base64 charsets ----------------------------+---------------------- Reporter: david.bianco | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: 2.0 Beta Resolution: fixed | Keywords: ----------------------------+---------------------- Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [8de3614afab68976fb683e433f95252e6ead4e3b/bro]: {{{ #!CommitTicketReference repository="bro" revision="8de3614afab68976fb683e433f95252e6ead4e3b" Merge remote-tracking branch 'origin/topic/jsiwek/custom-b64-alphabet' * origin/topic/jsiwek/custom-b64-alphabet: Add decode_base64_custom BiF to allow alternate base64 alphabets. Simplified the code a little bit. Closes #670. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 16 08:40:33 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 16 Nov 2011 16:40:33 -0000 Subject: [Bro-Dev] #674: topic/jsiwek/require-libmagic-libz In-Reply-To: <048.10e00f6a9511cdabaf15c7045c6e7a2d@tracker.bro-ids.org> References: <048.10e00f6a9511cdabaf15c7045c6e7a2d@tracker.bro-ids.org> Message-ID: <063.71a90d3beec6db202c4233cd6fd86757@tracker.bro-ids.org> #674: topic/jsiwek/require-libmagic-libz ----------------------------+------------------------ Reporter: jsiwek | Owner: robin Type: Merge Request | Status: closed Priority: Low | Milestone: Bro2.0 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [7696c8b3652d83aa737d04873c459495ae2caba8/bro]: {{{ #!CommitTicketReference repository="bro" revision="7696c8b3652d83aa737d04873c459495ae2caba8" Merge remote-tracking branch 'origin/topic/jsiwek/require-libmagic-libz' * origin/topic/jsiwek/require-libmagic-libz: Promote libz and libmagic to required dependencies. Conflicts: doc/quickstart.rst Closes #674 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 16 08:41:06 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 16 Nov 2011 16:41:06 -0000 Subject: [Bro-Dev] #666: broctl + macports python libraries In-Reply-To: <048.d70212220b0f1740e974880807144f64@tracker.bro-ids.org> References: <048.d70212220b0f1740e974880807144f64@tracker.bro-ids.org> Message-ID: <063.78f9670f70798a8a31c8d468dcd4d42b@tracker.bro-ids.org> #666: broctl + macports python libraries ----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => fixed Comment: In [b8eb146e0951773b2d90b58f5276a8b1867d263d/pysubnettree]: {{{ #!CommitTicketReference repository="pysubnettree" revision="b8eb146e0951773b2d90b58f5276a8b1867d263d" Merge remote-tracking branch 'origin/topic/jsiwek/find-pythondev' * origin/topic/jsiwek/find-pythondev: Simplify finding of Python headers/libraries. Closes #666. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 16 10:18:01 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 16 Nov 2011 18:18:01 -0000 Subject: [Bro-Dev] #679: Add support for deleting vector elements In-Reply-To: <050.0e994e46e1e8e7343905449869e3f2a1@tracker.bro-ids.org> References: <050.0e994e46e1e8e7343905449869e3f2a1@tracker.bro-ids.org> Message-ID: <065.65c85cdaf4e43d0290f2d44d6510b184@tracker.bro-ids.org> #679: Add support for deleting vector elements -----------------------+------------------------ Reporter: matthias | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: git/master Resolution: | Keywords: -----------------------+------------------------ Comment (by robin): On Tue, Nov 15, 2011 at 20:40 -0000, you wrote: > Sets and tables support the `delete` operation, and an intuitive > brogrammer expects the same thing to work with vectors as well. I'm still not sure about that: what would it do? Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 16 10:37:31 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 16 Nov 2011 18:37:31 -0000 Subject: [Bro-Dev] #679: Add support for deleting vector elements In-Reply-To: <050.0e994e46e1e8e7343905449869e3f2a1@tracker.bro-ids.org> References: <050.0e994e46e1e8e7343905449869e3f2a1@tracker.bro-ids.org> Message-ID: <065.5d1eae5027666847991c1fb542b92ceb@tracker.bro-ids.org> #679: Add support for deleting vector elements -----------------------+------------------------ Reporter: matthias | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: git/master Resolution: | Keywords: -----------------------+------------------------ Comment (by matthias): Replying to [comment:0 robin]: > I'm still not sure about that: what would it do? In this case, it would simple delete the 4th element in the vector (and perhaps even shrink it's size to 0). That is, before the delete we have: {{{ [ NULL, NULL, NULL, "foo" ] }}} and afterwards {{{ [ ] }}} Does that make sense? Matthias -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 16 10:47:53 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 16 Nov 2011 18:47:53 -0000 Subject: [Bro-Dev] #679: Add support for deleting vector elements In-Reply-To: <050.0e994e46e1e8e7343905449869e3f2a1@tracker.bro-ids.org> References: <050.0e994e46e1e8e7343905449869e3f2a1@tracker.bro-ids.org> Message-ID: <065.c58694860b42052e9ec8fc915a523ee5@tracker.bro-ids.org> #679: Add support for deleting vector elements -----------------------+------------------------ Reporter: matthias | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: git/master Resolution: | Keywords: -----------------------+------------------------ Comment (by robin): What does a vector do if I insert an element way beyond the current end? Does it expand and fill with null values in between? If so, I already don't like that, because Bro doesn't really have null values. Using one of those fields will probably cause a crash (or is it caught and causes a run-time error?) Is there a way to test if a vector value is set? (As you can tell, I have pretty much avoided using vectors in the past :-) One way to solve all this would be giving vectors a &default attribute to specify a value to be returned by default. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Wed Nov 16 11:16:03 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 16 Nov 2011 11:16:03 -0800 Subject: [Bro-Dev] Broxygen docs Message-ID: <20111116191603.GA57014@icir.org> (Moving this to bro-dev). I've only taken a quick look at your recent changes so far, but that looks good! Quick comments: - I think we should leave the "other components" out and document them separately on the web page. That stuff is pulled out of the other git repositories, and doing that manually (as I think one would have to, right?) is quite cumbersome and error prone. - The changes break the current web pages unfortunately, see: http://www.bro-ids.org/documentation-beta/index.html http://www.bro-ids.org/documentation-git/index.html We should be able to hard-code the beta page to the last commit before the merge, and can then keep working on the git version. - I suggest to use the non-serif font for the section headlines, and using serif only for the page title. That's how the web pages are doing as well and looks a bit better IMO. More later. I'll also start to adding some first documentation to event.bif to see how that gies. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From jsiwek at illinois.edu Wed Nov 16 11:44:30 2011 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Wed, 16 Nov 2011 19:44:30 +0000 Subject: [Bro-Dev] Broxygen docs In-Reply-To: <20111116191603.GA57014@icir.org> References: <20111116191603.GA57014@icir.org> Message-ID: <2A9B14A6-4CD3-484C-B5D4-EF7835F2B027@illinois.edu> > - I think we should leave the "other components" out and document > them separately on the web page. By letting Broxygen handle rendering of "other components", it seems to me that it makes it easier from a "documentation bundling" perspective in that you'll always get the component documentation that's associated with a given bro repo commit. > That stuff is pulled out of the > other git repositories, and doing that manually (as I think one > would have to, right?) is quite cumbersome and error prone. I think I'm not foreseeing the same process that you are. If 'www' delegates the rendering of "other components" to the 'bro' repo and that gets linked on the Broxygen main index, then all 'www' has to do is grab the 'html' output as a whole. Currently, there's nothing manual about getting Broxygen to render the correct "other components" because they're just symlinks into git submodules. > > - The changes break the current web pages unfortunately, see: > > http://www.bro-ids.org/documentation-beta/index.html > http://www.bro-ids.org/documentation-git/index.html > > We should be able to hard-code the beta page to the last commit > before the merge, and can then keep working on the git version. Yeah, I was kind of hoping for the Broxygen docs to completely replace the current documentation*/ directories. I mean you'd still have different versions of documentation but each one would be obtained my doing `make broxygen` for a particular commit in a 'bro' repo. Depending on your response to above issues, want me to take a shot ('www' intimidates me sometimes) at inserting the Broxygen docs? > - I suggest to use the non-serif font for the section headlines, > and using serif only for the page title. That's how the web > pages are doing as well and looks a bit better IMO. Ok. I think I saw a couple other style differences I can try to hunt down. +Jon From robin at icir.org Wed Nov 16 12:39:40 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 16 Nov 2011 12:39:40 -0800 Subject: [Bro-Dev] Broxygen docs In-Reply-To: <2A9B14A6-4CD3-484C-B5D4-EF7835F2B027@illinois.edu> References: <20111116191603.GA57014@icir.org> <2A9B14A6-4CD3-484C-B5D4-EF7835F2B027@illinois.edu> Message-ID: <20111116203940.GA74063@icir.org> On Wed, Nov 16, 2011 at 19:44 +0000, you wrote: > I think I'm not foreseeing the same process that you are. I may have just misunderstood how things are done currently. I didn't really look at it yet but thought I saw somewhere a comment about having to copy files over manually. Discard my concern if that's not the case. > has to do is grab the 'html' output as a whole. Currently, there's > nothing manual about getting Broxygen to render the correct "other > components" because they're just symlinks into git submodules. Even with this model (which is generally fine with me) we'll have one issue: the submodule docs will reflect whatever the specific revision is that the Bro supermodule pulls in. That's kind of right from the supermodule's perspective as that's what that version of the supermodule is using. But on www, we also list the modules separately and may want to document different versions there. Say we do a Bro release. The Bro docs are then frozen at that point and will show up on www/documentation. However, if we then did an intermediary Broccoli release, the Bro docs would keep showing the old version. Is there a way to avoid that when having the submodule docs in Sphinx as well? > Yeah, I was kind of hoping for the Broxygen docs to completely replace > the current documentation*/ directories. I mean you'd still have > different versions of documentation but each one would be obtained my > doing `make broxygen` for a particular commit in a 'bro' repo. Yes, that's my vision as well (or at least "replace *most* of the current doc directory" depending on how we decide to handle the submodules, per above). > Depending on your response to above issues, want me to take a shot > ('www' intimidates me sometimes) at inserting the Broxygen docs? That would great, yes. (www is starting to intimidate me as well as it's getting quite complex; moving more into Sphinx should help with that as well. Generally, if you find opportunity to clean things up or make simpler, feel free to go ahead. My main objective with www is just to keep things as automated as possible so that when we're doing updates to any of the repos, one doesn't always need to remember to go to www and do certain things there as well. That's where some of the complexity is coming from.). > Ok. I think I saw a couple other style differences I can try to hunt down. Thanks (but leaving things like this at lower priority is fine). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Wed Nov 16 12:53:56 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 16 Nov 2011 20:53:56 -0000 Subject: [Bro-Dev] #679: Add support for deleting vector elements In-Reply-To: <050.0e994e46e1e8e7343905449869e3f2a1@tracker.bro-ids.org> References: <050.0e994e46e1e8e7343905449869e3f2a1@tracker.bro-ids.org> Message-ID: <065.8d3334090015bcb3b923cf5641cbd4d5@tracker.bro-ids.org> #679: Add support for deleting vector elements -----------------------+------------------------ Reporter: matthias | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: git/master Resolution: | Keywords: -----------------------+------------------------ Comment (by matthias): Run this and you'll see weirdness: {{{ global v: vector of string; event bro_init() { print v; v[3] = "foo"; print v; print "-------------------"; local x = v[42]; print x; } }}} The output is: {{{ [] [, , , foo] ------------------- anonymous-function { if (Notice::ACTION_DROP in Notice::n$actions) ; } }}} Replying to [comment:3 robin]: > What does a vector do if I insert an element way beyond the current > end? Does it expand and fill with null values in between? If so, I > already don't like that, because Bro doesn't really have null values. Per the above example, yes, Bro expands the vector and fills it with NULL values. > Using one of those fields will probably cause a crash (or is it caught > and causes a run-time error?) Worse, you get a very weird value back that has nothing to do with the vector itself. > Is there a way to test if a vector value is set? (As you can tell, I have pretty much > avoided using vectors in the past :-) I'm not aware of any mechanism to test this. > One way to solve all this would be giving vectors a &default > attribute to specify a value to be returned by default. Trying this fails: {{{ global v: vector of string &default="foo"; }}} with the output: {{{ error in ./vector.bro, line 1: &default value has inconsistent type (foo and vector) fatal error in ./vector.bro, line 1: BroType::AsTableType (vector/table) (vector) }}} Weird enough? :-) -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at illinois.edu Wed Nov 16 12:59:12 2011 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Wed, 16 Nov 2011 20:59:12 +0000 Subject: [Bro-Dev] Broxygen docs In-Reply-To: <20111116203940.GA74063@icir.org> References: <20111116191603.GA57014@icir.org> <2A9B14A6-4CD3-484C-B5D4-EF7835F2B027@illinois.edu> <20111116203940.GA74063@icir.org> Message-ID: > I may have just misunderstood how things are done currently. I didn't > really look at it yet but thought I saw somewhere a comment about > having to copy files over manually. Discard my concern if that's not > the case. Ah, yeah. In a README I think I said you could copy files into the sphinx source tree to get it to render them since that's technically a way to do it. But the other way I said was just to symlink it and that's the way I currently did it. > Even with this model (which is generally fine with me) we'll have one > issue: the submodule docs will reflect whatever the specific revision > is that the Bro supermodule pulls in. That's kind of right from the > supermodule's perspective as that's what that version of the > supermodule is using. But on www, we also list the modules separately > and may want to document different versions there. Say we do a Bro > release. The Bro docs are then frozen at that point and will show up > on www/documentation. However, if we then did an intermediary Broccoli > release, the Bro docs would keep showing the old version. Is there a > way to avoid that when having the submodule docs in Sphinx as well? Ok, good point. I think that's going to have to be out of the scope of Sphinx/Broxygen. So we'll have two different ways of rendering "other components": 1) 'www' pulls in certain release revisions of each submodule to render on it's own page 2) 'bro' manages the rendering of submodule READMEs depending on whatever the current submodule pointers are, and then 'www' pulls this in as part of a versioned Bro documentation bundle +Jon From bro at tracker.bro-ids.org Wed Nov 16 18:25:48 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 17 Nov 2011 02:25:48 -0000 Subject: [Bro-Dev] #682: Beta Documentation Pages Broken Message-ID: <052.f99456ea45e245a71a918a6069c18336@tracker.bro-ids.org> #682: Beta Documentation Pages Broken ------------------------+--------------------------------- Reporter: maxfeldman | Type: Problem Status: new | Priority: Normal Milestone: | Component: Website Version: | Keywords: documentation, beta ------------------------+--------------------------------- http://bro-ids.org/documentation-beta/ seems to have various display errors, eg. " ERROR/3 (root/documentation-beta/__git__.index.bro.rst, line 6)" It also seems that many of the specific links are not found, eg. http ://bro-ids.org/documentation-beta/notice.bro.html -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 16 22:05:58 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 17 Nov 2011 06:05:58 -0000 Subject: [Bro-Dev] #683: Some BiFs should return a vector instead of a set/table Message-ID: <050.1a279086ae98a29669599697952475ed@tracker.bro-ids.org> #683: Some BiFs should return a vector instead of a set/table ----------------------+------------------------ Reporter: matthias | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.1 Component: Bro | Version: git/master Keywords: | ----------------------+------------------------ The following functions should have a `vector` as yield value rather than a set/table: {{{ split.*(...) find_all(...) }}} Moreover, `split` & friends have the yield value `table[count] of string` which starts at index 1. This is counterintuitive, as regular vectors start with a index at 0. I suggest replacing the yield value with `vector of string`. -- Ticket URL: Bro Tracker Bro Issue Tracker From vallentin at icir.org Wed Nov 16 22:47:21 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Wed, 16 Nov 2011 22:47:21 -0800 Subject: [Bro-Dev] Difference between port conversion functions Message-ID: <1321512225-sup-5501@samurai.local> The file bro.bif in the src directory contains two functions to convert count values into ports. function count_to_port%(c: count, t: transport_proto%): port %{ return new PortVal(c, (TransportProto)(t->InternalInt())); %} function to_port%(num: count, proto: transport_proto%): port %{ return new PortVal(num, (TransportProto)proto->AsEnum()); %} How do they differ and which one should be used? It looks like they do the same but have a slightly different implementation. If that's the case, can we remove one? Matthias From vern at icir.org Wed Nov 16 22:56:22 2011 From: vern at icir.org (Vern Paxson) Date: Wed, 16 Nov 2011 22:56:22 -0800 Subject: [Bro-Dev] Difference between port conversion functions In-Reply-To: <1321512225-sup-5501@samurai.local> (Wed, 16 Nov 2011 22:47:21 PST). Message-ID: <20111117065622.2130C2C4003@rock.ICSI.Berkeley.EDU> > How do they differ and which one should be used? FWIW, darned if I know why there are two. Vern From robin at icir.org Thu Nov 17 07:43:47 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 17 Nov 2011 07:43:47 -0800 Subject: [Bro-Dev] Difference between port conversion functions In-Reply-To: <1321512225-sup-5501@samurai.local> References: <1321512225-sup-5501@samurai.local> Message-ID: <20111117154346.GA48386@icir.org> On Wed, Nov 16, 2011 at 22:47 -0800, you wrote: > function count_to_port%(c: count, t: transport_proto%): port > function to_port%(num: count, proto: transport_proto%): port Yeah, looks like one can go. We should keep the one which's naming is more consistent with other coverters; which I believe should be count_to_port because the first argument is a specific type (and not "any" as other to_* function use). This is out of my memory though, worth checking. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Thu Nov 17 07:59:59 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 17 Nov 2011 15:59:59 -0000 Subject: [Bro-Dev] #682: Beta Documentation Pages Broken In-Reply-To: <052.f99456ea45e245a71a918a6069c18336@tracker.bro-ids.org> References: <052.f99456ea45e245a71a918a6069c18336@tracker.bro-ids.org> Message-ID: <067.b0a478ec9ba967ca8131072f370782da@tracker.bro-ids.org> #682: Beta Documentation Pages Broken -------------------------+--------------------------------- Reporter: maxfeldman | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Component: Website | Version: Resolution: | Keywords: documentation, beta -------------------------+--------------------------------- Changes (by robin): * status: new => closed Comment: We're reworking the structure of the docs but this shouldn't have been visible. Fixed now, thanks! -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org -- Ticket URL: Bro Tracker Bro Issue Tracker From vallentin at icir.org Thu Nov 17 08:06:54 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Thu, 17 Nov 2011 08:06:54 -0800 Subject: [Bro-Dev] Difference between port conversion functions In-Reply-To: <20111117154346.GA48386@icir.org> References: <1321512225-sup-5501@samurai.local> <20111117154346.GA48386@icir.org> Message-ID: <1321545844-sup-2104@samurai.local> > > function count_to_port%(c: count, t: transport_proto%): port > > function to_port%(num: count, proto: transport_proto%): port > > Yeah, looks like one can go. We should keep the one which's naming is > more consistent with other coverters; which I believe should be > count_to_port because the first argument is a specific type (and not > "any" as other to_* function use). This is out of my memory though, > worth checking. I double-checked it and it is indeed the case. In fact, the to_* functions are all string converters and (should) have a signature of the form to_TYPE(s: string): TYPE As such, to_port(...) would have to be changed to to_port(s: string) : port if we want to be consistent. Matthias From robin at icir.org Thu Nov 17 08:14:04 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 17 Nov 2011 08:14:04 -0800 Subject: [Bro-Dev] Broxygen docs In-Reply-To: References: <20111116191603.GA57014@icir.org> <2A9B14A6-4CD3-484C-B5D4-EF7835F2B027@illinois.edu> <20111116203940.GA74063@icir.org> Message-ID: <20111117161404.GJ48386@icir.org> On Wed, Nov 16, 2011 at 20:59 +0000, you wrote: > 1) 'www' pulls in certain release revisions of each submodule to > render on it's own page > 2) 'bro' manages the rendering of submodule READMEs depending on > whatever the current submodule pointers are, and then 'www' pulls this > in as part of a versioned Bro documentation bundle Yeah, that makes sense. We'll need to keep in mind to not use Sphinx features in the READMEs so that they still render standalone. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Thu Nov 17 09:48:26 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 17 Nov 2011 17:48:26 -0000 Subject: [Bro-Dev] #684: Change to_port BiF to take a string as argument Message-ID: <050.b20757078e176ef09b16f1693c6f73e6@tracker.bro-ids.org> #684: Change to_port BiF to take a string as argument ----------------------+------------------------ Reporter: matthias | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.1 Component: Bro | Version: git/master Keywords: | ----------------------+------------------------ The function `to_port`, which is currently defined as {{{ function to_port%(num: count, proto: transport_proto%): port %{ return new PortVal(num, (TransportProto)proto->AsEnum()); %} }}} should have a different signature in order to be consistent with the other `to_*` conversion functions. The new signature should be: {{{ function to_port%(s: string%): port; }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Nov 17 11:58:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 17 Nov 2011 19:58:13 -0000 Subject: [Bro-Dev] #679: Add support for deleting vector elements In-Reply-To: <050.0e994e46e1e8e7343905449869e3f2a1@tracker.bro-ids.org> References: <050.0e994e46e1e8e7343905449869e3f2a1@tracker.bro-ids.org> Message-ID: <065.388f7f9dc0e0819a48f9945d8c0b6860@tracker.bro-ids.org> #679: Add support for deleting vector elements -----------------------+------------------------ Reporter: matthias | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: git/master Resolution: | Keywords: -----------------------+------------------------ Comment (by gregor): > Per the above example, yes, Bro expands the vector and fills it with NULL > values. > > > Using one of those fields will probably cause a crash (or is it caught > > and causes a run-time error?) > > Worse, you get a very weird value back that has nothing to do with the > vector itself. What happens if you access v[2]?? The other (potentially independent) question is what v[42] does. Is the vector extended to contain 43 elements or does the core just access a non- existent element and thus just gets some random data back? Your output looks like its the latter case. Maybe the goal for 2.1 should be to do a complete overhaul/redesign of list / vector functionality in Bro. I.e., lets discuss what functionality makes sense and then implement that. IIRC, the current functionality is inconsistent and partially broken. And I think vectors are hardly used, so it shouldn't be too painful to adapt script level code. cu Gregor -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Thu Nov 17 12:09:56 2011 From: gregor at icir.org (Gregor Maier) Date: Thu, 17 Nov 2011 12:09:56 -0800 Subject: [Bro-Dev] Difference between port conversion functions In-Reply-To: <1321512225-sup-5501@samurai.local> References: <1321512225-sup-5501@samurai.local> Message-ID: <4EC56A14.1020905@icir.org> > %{ > return new PortVal(num, (TransportProto)proto->AsEnum()); > %} We should probably use the above implementation though, because AFAIK AsEnum() does type checking whereas InternalInt() won't. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Thu Nov 17 14:17:51 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 17 Nov 2011 22:17:51 -0000 Subject: [Bro-Dev] #679: Add support for deleting vector elements In-Reply-To: <050.0e994e46e1e8e7343905449869e3f2a1@tracker.bro-ids.org> References: <050.0e994e46e1e8e7343905449869e3f2a1@tracker.bro-ids.org> Message-ID: <065.cc4165153618f20eb81cef61d82084df@tracker.bro-ids.org> #679: Add support for deleting vector elements -----------------------+------------------------ Reporter: matthias | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: | Keywords: language -----------------------+------------------------ Changes (by robin): * keywords: => language * milestone: Bro2.0 => Comment: Right now I don't really see this all for 2.1 (and also not for 2.0 as the milestone says :). But I'm tagging it with `language` to group it with the other script language related stuff. Please use that tag for other related tickets as well. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Nov 17 15:05:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 17 Nov 2011 23:05:28 -0000 Subject: [Bro-Dev] #679: Add support for deleting vector elements In-Reply-To: <050.0e994e46e1e8e7343905449869e3f2a1@tracker.bro-ids.org> References: <050.0e994e46e1e8e7343905449869e3f2a1@tracker.bro-ids.org> Message-ID: <065.cfd2893abe97f039a6bec9c04acb34cd@tracker.bro-ids.org> #679: Add support for deleting vector elements -----------------------+------------------------ Reporter: matthias | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: | Keywords: language -----------------------+------------------------ Comment (by matthias): Replying to [comment:5 gregor]: > What happens if you access v[2]?? You mean this? {{{ global v: vector of string; event bro_init() { local a = v[2]; v[3] = "bar-baz"; local b = v[2]; print "-------------------"; print a; print "-------------------"; print b; print "-------------------"; } }}} It prints: {{{ ------------------- anonymous-function { if (Notice::ACTION_DROP in Notice::n$actions) ; } ------------------- 22/tcp ------------------- }}} > Maybe the goal for 2.1 should be to do a complete overhaul/redesign of list / vector functionality in Bro. I.e., lets discuss what functionality makes sense and then implement that. IIRC, the current functionality is inconsistent and partially broken. And I think vectors are hardly used, so it shouldn't be too painful to adapt script level code. I would really like to see that, but as Robin mentions, maybe at some future point in time. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Thu Nov 17 15:24:27 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 17 Nov 2011 15:24:27 -0800 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/jsiwek/broxygen-cleanup: Teach Broxygen :bro:see: role for referencing any identifier in Bro domain. (5227eb7) In-Reply-To: <201111172256.pAHMurPm029255@bro-ids.icir.org> References: <201111172256.pAHMurPm029255@bro-ids.icir.org> Message-ID: <20111117232427.GF69499@icir.org> On Thu, Nov 17, 2011 at 14:56 -0800, Jonathan Siwek wrote: > Teach Broxygen :bro:see: role for referencing any identifier in Bro domain. Very cool! Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Thu Nov 17 15:30:01 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 17 Nov 2011 23:30:01 -0000 Subject: [Bro-Dev] #685: Change default constant for empty or unset field of LogWriterAscii Message-ID: <048.4a3ac67b40f5405235e87970b76f67ce@tracker.bro-ids.org> #685: Change default constant for empty or unset field of LogWriterAscii --------------------+--------------------- Reporter: amannb | Type: Problem Status: new | Priority: Normal Milestone: | Component: Bro Version: | Keywords: --------------------+--------------------- At the moment, the ascii logwriter uses the same string to represent empty fields and unset fields. Thus, when reading the logfiles later, for sets and vectors one cannot distinguish if the set/vector was unset or empty. Changing either one of the empty_field or unset_field values would solve the problem. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Nov 17 16:39:35 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 18 Nov 2011 00:39:35 -0000 Subject: [Bro-Dev] #686: Python linker path problem Message-ID: <047.e282692296a66ebf926ac014b9c39475@tracker.bro-ids.org> #686: Python linker path problem ---------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Keywords: | ---------------------+-------------------- This is on FreeBSD 8.1 with the python27 port installed and current master: {{{ Scanning dependencies of target _SubnetTree [ 76%] Building CXX object aux/broctl/aux/pysubnettree/CMakeFiles/_SubnetTree.dir/SubnetTreePYTHON_wrap.cxx.o [ 76%] Building CXX object aux/broctl/aux/pysubnettree/CMakeFiles/_SubnetTree.dir/SubnetTree.cc.o [ 76%] Building C object aux/broctl/aux/pysubnettree/CMakeFiles/_SubnetTree.dir/patricia.c.o Linking CXX shared module _SubnetTree.so /usr/bin/ld: cannot find -lpython2.7 *** Error code 1 }}} Adding `-L/usr/local/lib` to `LDFLAGS` solves the problem. {{{ # python --version Python 2.7.2 # for i in --prefix --exec-prefix --includes --libs --cflags --ldflags; do python-config $i; done /usr/local /usr/local -I/usr/local/include/python2.7 -I/usr/local/include/python2.7 -lutil -lm -lpython2.7 -I/usr/local/include/python2.7 -I/usr/local/include/python2.7 -fno-strict- aliasing -O2 -pipe -fno-strict-aliasing -DNDEBUG -O2 -pipe -fno-strict- aliasing -L/usr/local/lib/python2.7/config -lutil -lm -lpython2.7 -Wl,--export- dynamic # ls -al /usr/local/lib/libpython* -r-xr-xr-x 1 root wheel 2914926 Nov 17 14:12 /usr/local/lib/libpython2.7.a lrwxr-xr-x 1 root wheel 17 Nov 17 14:13 /usr/local/lib/libpython2.7.so -> libpython2.7.so.1 -r-xr-xr-x 1 root wheel 1682880 Nov 17 14:13 /usr/local/lib/libpython2.7.so.1 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Thu Nov 17 16:44:15 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 17 Nov 2011 16:44:15 -0800 Subject: [Bro-Dev] Broxygen feature request Message-ID: <20111118004415.GG69499@icir.org> Sorry, I have yet another feature request for Broxygen ... Working on event.bif, I was thinking it would be nice if I could group events somehow. Currently, they are all listed alphabetically. If there were markup to define a group (a directive? role? other magic?) we could, e.g., put all HTTP events into their own section "HTTP Analyzer". Doable? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Thu Nov 17 16:57:44 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 18 Nov 2011 00:57:44 -0000 Subject: [Bro-Dev] #687: Smaller make trouble Message-ID: <047.35a104e568a47473cb847c4603f4a3eb@tracker.bro-ids.org> #687: Smaller make trouble ---------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Keywords: | ---------------------+-------------------- On FreeBSD 8.1 after `make distclean`: {{{ LDFLAGS=-L/usr/local/lib ./configure --prefix=$HOME && make -j 8 [?] GeoIP: true Google perftools: false ================================================================ -- Configuring done -- Generating done -- Build files have been written to: /usr/home/bro/src/bro-git/build `build' is up to date. }}} I.e., it doesn't build. Typing `make` then works though. This is reproducible, but oddly if I do the steps configure and make separately, it works just fine. Any idea? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Nov 17 16:59:30 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 18 Nov 2011 00:59:30 -0000 Subject: [Bro-Dev] #687: Smaller make trouble In-Reply-To: <047.35a104e568a47473cb847c4603f4a3eb@tracker.bro-ids.org> References: <047.35a104e568a47473cb847c4603f4a3eb@tracker.bro-ids.org> Message-ID: <062.187be7f3fbbbdab2105cde18c2c576f4@tracker.bro-ids.org> #687: Smaller make trouble ----------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by robin): Next problem: when I type `make instal`, it doesn't do anything: {{{ # make install make -C build [ 0%] Built target binpac_lib [ 13%] Built target binpac [ 15%] Built target bifcl [ 76%] Built target bro [ 76%] Built target _SubnetTree [ 76%] Built target capstats [ 76%] Built target adtrace [ 77%] Built target ftwire2bro [ 77%] Built target nfcollector [ 79%] Built target rst [ 87%] Built target broccoli [ 95%] Built target broccoliStatic [ 97%] Built target broconftest [ 97%] Built target broconn [ 97%] Built target broenum [ 97%] Built target brohose [ 98%] Built target broping [ 98%] Built target brotable [100%] Built target _broccoli_intern [100%] Built target broccoli_ext }}} Doing the same in `build` works. Could this be some non-GNU make problem triggered by the recent changes to the top-level Makefile? -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Thu Nov 17 17:03:57 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 17 Nov 2011 17:03:57 -0800 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/script-reference: In-Reply-To: <201111180100.pAI10gil027308@bro-ids.icir.org> References: <201111180100.pAI10gil027308@bro-ids.icir.org> Message-ID: <20111118010357.GH69499@icir.org> On Thu, Nov 17, 2011 at 17:00 -0800, I wrote: > Repository : ssh://git at bro-ids.icir.org/bro > > On branch : topic/script-reference Created this branch for us all to work on Broxygen comments. It's branched off of master so Jon's recent changes aren't in there yet. I've started with a few handlers in event.bif, still trying to find the right style. These are just place-holders for now: > +## see:: *connection* Will replace them later. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From vallentin at icir.org Thu Nov 17 18:43:44 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Thu, 17 Nov 2011 18:43:44 -0800 Subject: [Bro-Dev] Broxygen feature request In-Reply-To: <20111118004415.GG69499@icir.org> References: <20111118004415.GG69499@icir.org> Message-ID: <1321584061-sup-4122@samurai.local> > Working on event.bif, I was thinking it would be nice if I could group > events somehow. Currently, they are all listed alphabetically. If > there were markup to define a group (a directive? role? other magic?) > we could, e.g., put all HTTP events into their own section "HTTP > Analyzer". Isn't that related to the &group="x" attribute? If we use it universally, Broxygen could just use it for this purpose. Matthias From robin at icir.org Fri Nov 18 06:58:49 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 18 Nov 2011 06:58:49 -0800 Subject: [Bro-Dev] Broxygen feature request In-Reply-To: <1321584061-sup-4122@samurai.local> References: <20111118004415.GG69499@icir.org> <1321584061-sup-4122@samurai.local> Message-ID: <20111118145849.GC24022@icir.org> On Thu, Nov 17, 2011 at 18:43 -0800, you wrote: > Isn't that related to the &group="x" attribute? That could work if we also had a mapping from that group to some nicer section title. On the other hand, grouping might also be usefol for other things than events (like the options in init-bare.bro). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Fri Nov 18 07:00:46 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 18 Nov 2011 15:00:46 -0000 Subject: [Bro-Dev] #685: Change default constant for empty or unset field of LogWriterAscii In-Reply-To: <048.4a3ac67b40f5405235e87970b76f67ce@tracker.bro-ids.org> References: <048.4a3ac67b40f5405235e87970b76f67ce@tracker.bro-ids.org> Message-ID: <063.1e90855488b0e9c0f0921efd5c9beda7@tracker.bro-ids.org> #685: Change default constant for empty or unset field of LogWriterAscii ----------------------+-------------------- Reporter: amannb | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Changes (by robin): * milestone: => Bro2.0 Comment: > Changing either one of the empty_field or unset_field values would solve > the problem. That makes sense, we should have used different values from the beginning to make the cases distinguishable. I'm thinking we should change that for 2.0 before people get used to the new format. Opinions? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Nov 18 07:21:41 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 18 Nov 2011 15:21:41 -0000 Subject: [Bro-Dev] #685: Change default constant for empty or unset field of LogWriterAscii In-Reply-To: <048.4a3ac67b40f5405235e87970b76f67ce@tracker.bro-ids.org> References: <048.4a3ac67b40f5405235e87970b76f67ce@tracker.bro-ids.org> Message-ID: <063.7f225e4e643eecb7f22faf9a3a559b25@tracker.bro-ids.org> #685: Change default constant for empty or unset field of LogWriterAscii ----------------------+-------------------- Reporter: amannb | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by seth): Tough question. Now that we have bro-cut and people can start processing logs with that and it understands the log format implicitly I think we should make empty fields empty. NULL fields can still be "-". What about that? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Nov 18 07:55:03 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 18 Nov 2011 15:55:03 -0000 Subject: [Bro-Dev] #685: Change default constant for empty or unset field of LogWriterAscii In-Reply-To: <048.4a3ac67b40f5405235e87970b76f67ce@tracker.bro-ids.org> References: <048.4a3ac67b40f5405235e87970b76f67ce@tracker.bro-ids.org> Message-ID: <063.555025ca5ae6dd451c980f8ce42afd40@tracker.bro-ids.org> #685: Change default constant for empty or unset field of LogWriterAscii ----------------------+-------------------- Reporter: amannb | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by robin): On Fri, Nov 18, 2011 at 15:21 -0000, you wrote: > Tough question. Now that we have bro-cut and people can start processing > logs with that and it understands the log format implicitly I think we > should make empty fields empty. NULL fields can still be "-". What about > that? Not sure I like that. That would indeed work well with bro-cut, but it make the logs really hard to read visually. Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Nov 18 08:37:00 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 18 Nov 2011 16:37:00 -0000 Subject: [Bro-Dev] #686: Python linker path problem In-Reply-To: <047.e282692296a66ebf926ac014b9c39475@tracker.bro-ids.org> References: <047.e282692296a66ebf926ac014b9c39475@tracker.bro-ids.org> Message-ID: <062.68a6f86cec247e1ca6e5e6d3c71b03c6@tracker.bro-ids.org> #686: Python linker path problem -----------------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by jsiwek): * status: new => closed * resolution: => Solved/Applied Comment: This should be fixed after merging fastpath commit [62ad522d1dce638653a0221a4ea2be7b4cb102d2/cmake] and updating submodules. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Nov 18 09:46:30 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 18 Nov 2011 17:46:30 -0000 Subject: [Bro-Dev] #687: Smaller make trouble In-Reply-To: <047.35a104e568a47473cb847c4603f4a3eb@tracker.bro-ids.org> References: <047.35a104e568a47473cb847c4603f4a3eb@tracker.bro-ids.org> Message-ID: <062.de13aa71453fb7238ae2d5e587f181d1@tracker.bro-ids.org> #687: Smaller make trouble ----------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by jsiwek): > Could this be some non-GNU make problem triggered by the recent changes to the top-level Makefile? Yes, looks like it. Try after merging `topic/jsiwek/parallel-make- recursion` again in `bro`, `binpac`, `bro-aux`, `broccoli`, `broctl`, and `capstats` repos. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Nov 18 09:47:42 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 18 Nov 2011 17:47:42 -0000 Subject: [Bro-Dev] #687: Smaller make trouble In-Reply-To: <047.35a104e568a47473cb847c4603f4a3eb@tracker.bro-ids.org> References: <047.35a104e568a47473cb847c4603f4a3eb@tracker.bro-ids.org> Message-ID: <062.f703027a91630f54095d8b79f80816ea@tracker.bro-ids.org> #687: Smaller make trouble ----------------------------+-------------------- Reporter: robin | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------------+-------------------- Changes (by jsiwek): * type: Problem => Merge Request -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at illinois.edu Fri Nov 18 09:54:20 2011 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Fri, 18 Nov 2011 17:54:20 +0000 Subject: [Bro-Dev] Broxygen feature request In-Reply-To: <20111118004415.GG69499@icir.org> References: <20111118004415.GG69499@icir.org> Message-ID: <96A2C639-68DD-47B6-8F3B-FE2245FC8A9B@illinois.edu> > Working on event.bif, I was thinking it would be nice if I could group > events somehow. Currently, they are all listed alphabetically. If > there were markup to define a group (a directive? role? other magic?) > we could, e.g., put all HTTP events into their own section "HTTP > Analyzer". Where would you want the grouping to occur in the rendered docs? In a custom index, inline in the events.bif docs, somewhere else? > Doable? Yes, but I don't currently have a simple solution (or any solution) in mind yet. +Jon From robin at icir.org Fri Nov 18 10:00:15 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 18 Nov 2011 10:00:15 -0800 Subject: [Bro-Dev] Broxygen feature request In-Reply-To: <96A2C639-68DD-47B6-8F3B-FE2245FC8A9B@illinois.edu> References: <20111118004415.GG69499@icir.org> <96A2C639-68DD-47B6-8F3B-FE2245FC8A9B@illinois.edu> Message-ID: <20111118180015.GE34647@icir.org> On Fri, Nov 18, 2011 at 17:54 +0000, you wrote: > Where would you want the grouping to occur in the rendered docs? In a > custom index, inline in the events.bif docs, somewhere else? I was thinking inline in events.bif, ideally with a toc at the beginning. But perhaps it then wouldn't really be events.bif any more from the perspective of someone reading the docs; but just a section in the documentation called "Built-in Events", with corresponding subsections. > Yes, but I don't currently have a simple solution (or any solution) in mind yet. It's not so important, I haven't really thought it through how it would fit into the overall structure. It's something we can always add later. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Fri Nov 18 11:15:18 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 18 Nov 2011 11:15:18 -0800 Subject: [Bro-Dev] content_gap vs. ack_above_hole Message-ID: <20111118191518.GG34647@icir.org> Can somebody remind me what exactly the difference between these two is (and/or why we have both?). >From a quick look at the code, they both report data being acked that's not seen, but with slightly different specifics. I'm not sure what a good way is to describe the one event vs the other. Robin, writing Broxygen docs. :) -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Fri Nov 18 11:23:33 2011 From: seth at icir.org (Seth Hall) Date: Fri, 18 Nov 2011 14:23:33 -0500 Subject: [Bro-Dev] content_gap vs. ack_above_hole In-Reply-To: <20111118191518.GG34647@icir.org> References: <20111118191518.GG34647@icir.org> Message-ID: <36F13567-7403-450D-9F46-513BCA0CA2DB@icir.org> On Nov 18, 2011, at 2:15 PM, Robin Sommer wrote: > Can somebody remind me what exactly the difference between these two > is (and/or why we have both?). I've wondered the same thing for a long time and can't wait to find out. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From vern at icir.org Fri Nov 18 11:26:04 2011 From: vern at icir.org (Vern Paxson) Date: Fri, 18 Nov 2011 11:26:04 -0800 Subject: [Bro-Dev] content_gap vs. ack_above_hole In-Reply-To: <20111118191518.GG34647@icir.org> (Fri, 18 Nov 2011 11:15:18 PST). Message-ID: <20111118192604.1FE182C4005@rock.ICSI.Berkeley.EDU> > Can somebody remind me what exactly the difference between these two > is (and/or why we have both?). Yeah, my fault :-P. As best as I can tell (from revisiting the code), content-gap is a superset of ack-above-hole. Content gaps can also occur in situations where we're not expecting to see ACKs (for example, due to split routing, or because we're not processing traffic from the receiver). I think merging the two into a single content_gap event would make sense. Vern From robin at icir.org Fri Nov 18 11:30:39 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 18 Nov 2011 11:30:39 -0800 Subject: [Bro-Dev] content_gap vs. ack_above_hole In-Reply-To: <20111118192604.1FE182C4005@rock.ICSI.Berkeley.EDU> References: <20111118191518.GG34647@icir.org> <20111118192604.1FE182C4005@rock.ICSI.Berkeley.EDU> Message-ID: <20111118193039.GH34647@icir.org> On Fri, Nov 18, 2011 at 11:26 -0800, you wrote: > content-gap is a superset of ack-above-hole. (... and to add some confusion, the gap_report event is actually tied closely to ack_above_hole, not to content_gap.) > processing traffic from the receiver). I think merging the two into a > single content_gap event would make sense. Ok, something for the todo list. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Fri Nov 18 11:34:53 2011 From: seth at icir.org (Seth Hall) Date: Fri, 18 Nov 2011 14:34:53 -0500 Subject: [Bro-Dev] Bro's snap length Message-ID: <88ACE3F3-553F-4A35-BB3F-FC47FC80A8EC@icir.org> We're going to need to change Bro's default snap length before the 2.0 final release or at least do something. I've run into several people now who are having really abysmal performance (dropping packets at relatively low data rate) and when they run with a reduced snaplen the performance immediately improves. If anyone is seeing apparent performance problems with Bro, add the following line to broctl.cfg: broargs=-l 9800 You can set the value to match your MTU but you also need to be aware that you must disable some offload features of NICs. More information about the issue can be found at a recent post on the security onion blog. http://securityonion.blogspot.com/2011/10/when-is-full-packet-capture-not-full.html .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Fri Nov 18 12:08:16 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 18 Nov 2011 12:08:16 -0800 Subject: [Bro-Dev] Bro's snap length In-Reply-To: <88ACE3F3-553F-4A35-BB3F-FC47FC80A8EC@icir.org> References: <88ACE3F3-553F-4A35-BB3F-FC47FC80A8EC@icir.org> Message-ID: <20111118200816.GA38820@icir.org> On Fri, Nov 18, 2011 at 14:34 -0500, you wrote: > We're going to need to change Bro's default snap length before the 2.0 > final release or at least do something. I think we should add a script-level option snaplen and set it to a much smaller default, like 8K or 4K. Plus a FAQ entry, linked from the Quickstart guide, explaining what's going on and what to do. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From jones at tacc.utexas.edu Fri Nov 18 12:48:44 2011 From: jones at tacc.utexas.edu (William Jones) Date: Fri, 18 Nov 2011 20:48:44 +0000 Subject: [Bro-Dev] Bro's snap length In-Reply-To: <88ACE3F3-553F-4A35-BB3F-FC47FC80A8EC@icir.org> References: <88ACE3F3-553F-4A35-BB3F-FC47FC80A8EC@icir.org> Message-ID: What type of ether net card are they using. Is pf_ring nabled? Bill Jones -----Original Message----- From: bro-dev-bounces at bro-ids.org [mailto:bro-dev-bounces at bro-ids.org] On Behalf Of Seth Hall Sent: Friday, November 18, 2011 1:35 PM To: bro-dev at bro-ids.org Dev Subject: [Bro-Dev] Bro's snap length We're going to need to change Bro's default snap length before the 2.0 final release or at least do something. I've run into several people now who are having really abysmal performance (dropping packets at relatively low data rate) and when they run with a reduced snaplen the performance immediately improves. If anyone is seeing apparent performance problems with Bro, add the following line to broctl.cfg: broargs=-l 9800 You can set the value to match your MTU but you also need to be aware that you must disable some offload features of NICs. More information about the issue can be found at a recent post on the security onion blog. http://securityonion.blogspot.com/2011/10/when-is-full-packet-capture-not-full.html .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ _______________________________________________ bro-dev mailing list bro-dev at bro-ids.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From seth at icir.org Fri Nov 18 12:53:18 2011 From: seth at icir.org (Seth Hall) Date: Fri, 18 Nov 2011 15:53:18 -0500 Subject: [Bro-Dev] Bro's snap length In-Reply-To: References: <88ACE3F3-553F-4A35-BB3F-FC47FC80A8EC@icir.org> Message-ID: On Nov 18, 2011, at 3:48 PM, William Jones wrote: > What type of ether net card are they using. > Is pf_ring nabled? This is a separate issue that shows up on almost every NIC but I don't know what wrinkles PF_RING introduces. I suppose it could also depend on if you are using PF_RING's DNA drivers or not since I believe that's doing direct memory injection from the NIC which changes the behavior where we are currently seeing trouble. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Fri Nov 18 14:21:37 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 18 Nov 2011 22:21:37 -0000 Subject: [Bro-Dev] #650: Finish Broxygen docs In-Reply-To: <047.9abf8a0851b3b85991457973560c772f@tracker.bro-ids.org> References: <047.9abf8a0851b3b85991457973560c772f@tracker.bro-ids.org> Message-ID: <062.84ef835471f4c4fc0d7768bd5664d0e1@tracker.bro-ids.org> #650: Finish Broxygen docs ----------------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Merge Request | Status: reopened Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------------+---------------------- Changes (by jsiwek): * status: closed => reopened * resolution: fixed => Comment: I put some more finishing touches in the `topic/jsiwek/broxygen-cleanup` branch: * style tweaks (seems to mimic website pretty well) * an index of notices gets automatically generated * a new :bro:see: role for xref'ing any identifier in the bro domain (I'm going to skip the .. bro:see:: directive for now, it was giving me trouble) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Nov 18 14:25:03 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 18 Nov 2011 22:25:03 -0000 Subject: [Bro-Dev] #650: Finish Broxygen docs In-Reply-To: <047.9abf8a0851b3b85991457973560c772f@tracker.bro-ids.org> References: <047.9abf8a0851b3b85991457973560c772f@tracker.bro-ids.org> Message-ID: <062.10a90e97880714e23148c40ba44d50e2@tracker.bro-ids.org> #650: Finish Broxygen docs ----------------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Merge Request | Status: reopened Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------------+---------------------- Comment (by robin): On Fri, Nov 18, 2011 at 22:21 -0000, you wrote: > * a new :bro:see: role for xref'ing any identifier in the bro domain (I'm > going to skip the .. bro:see:: directive for now, it was giving me > trouble) What's the problem? Having written a few event docstrings now, I think having the directive would be quite convinient, compared to having to write out a number of roles in sequence; and it would enforce a similar style. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Nov 18 14:25:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 18 Nov 2011 22:25:50 -0000 Subject: [Bro-Dev] #688: [Fwd] Re: content_gap vs. ack_above_hole Message-ID: <047.b7dcdeb9f90d2bebf224ed22fe455fa7@tracker.bro-ids.org> #688: [Fwd] Re: [Bro-Dev] content_gap vs. ack_above_hole ---------------------+----------------------- Reporter: robin | Type: Problem Status: new | Priority: Normal Component: Bro | Version: git/master Keywords: cleanup | ---------------------+----------------------- ----- Forwarded message from Vern Paxson ----- Date: Fri, 18 Nov 2011 11:26:04 -0800 From: Vern Paxson Subject: Re: [Bro-Dev] content_gap vs. ack_above_hole Message-Id: <20111118192604.1FE182C4005 at rock.ICSI.Berkeley.EDU> > Can somebody remind me what exactly the difference between these two > is (and/or why we have both?). Yeah, my fault :-P. As best as I can tell (from revisiting the code), content-gap is a superset of ack-above-hole. Content gaps can also occur in situations where we're not expecting to see ACKs (for example, due to split routing, or because we're not processing traffic from the receiver). I think merging the two into a single content_gap event would make sense. Vern ----- End forwarded message ----- -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Nov 18 14:28:22 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 18 Nov 2011 22:28:22 -0000 Subject: [Bro-Dev] #688: [Fwd] Re: content_gap vs. ack_above_hole In-Reply-To: <047.b7dcdeb9f90d2bebf224ed22fe455fa7@tracker.bro-ids.org> References: <047.b7dcdeb9f90d2bebf224ed22fe455fa7@tracker.bro-ids.org> Message-ID: <062.645abf5f5b45c856661d7ccd71345930@tracker.bro-ids.org> #688: [Fwd] Re: [Bro-Dev] content_gap vs. ack_above_hole ---------------------+------------------------ Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: | Keywords: cleanup ---------------------+------------------------ Changes (by robin): * type: Problem => Task Old description: > ----- Forwarded message from Vern Paxson ----- > > Date: Fri, 18 Nov 2011 11:26:04 -0800 > From: Vern Paxson > Subject: Re: [Bro-Dev] content_gap vs. ack_above_hole > Message-Id: <20111118192604.1FE182C4005 at rock.ICSI.Berkeley.EDU> > > > Can somebody remind me what exactly the difference between these two > > is (and/or why we have both?). > > Yeah, my fault :-P. As best as I can tell (from revisiting the code), > content-gap is a superset of ack-above-hole. Content gaps can also occur > in situations where we're not expecting to see ACKs (for example, due to > split routing, or because we're not processing traffic from the > receiver). > I think merging the two into a single content_gap event would make sense. > > Vern > > ----- End forwarded message ----- New description: From: Vern Paxson Subject: Re: [Bro-Dev] content_gap vs. ack_above_hole > Can somebody remind me what exactly the difference between these two > is (and/or why we have both?). Yeah, my fault :-P. As best as I can tell (from revisiting the code), content-gap is a superset of ack-above-hole. Content gaps can also occur in situations where we're not expecting to see ACKs (for example, due to split routing, or because we're not processing traffic from the receiver). I think merging the two into a single content_gap event would make sense. Vern -- -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Nov 18 14:39:15 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 18 Nov 2011 22:39:15 -0000 Subject: [Bro-Dev] #650: Finish Broxygen docs In-Reply-To: <047.9abf8a0851b3b85991457973560c772f@tracker.bro-ids.org> References: <047.9abf8a0851b3b85991457973560c772f@tracker.bro-ids.org> Message-ID: <062.fc772367cc37115126111530e3adfbd4@tracker.bro-ids.org> #650: Finish Broxygen docs ----------------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Merge Request | Status: reopened Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------------+---------------------- Comment (by jsiwek): > What's the problem? Mostly that Sphinx wasn't doing what I expected it to do or that I don't understand how to get it to work. > Having written a few event docstrings now, I think having the > directive would be quite convinient, compared to having to write out a > number of roles in sequence; and it would enforce a similar style. Ok. I'll look at it again. -- Ticket URL: Bro Tracker Bro Issue Tracker From vallentin at icir.org Fri Nov 18 19:14:51 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Fri, 18 Nov 2011 19:14:51 -0800 Subject: [Bro-Dev] Some bro.bif issues Message-ID: <1321665790-sup-8037@samurai.local> This is the function set_buf: function set_buf%(f: file, buffered: bool%): any %{ f->SetBuf(buffered); return new Val(0, TYPE_VOID); %} It's return value is any although void is returned. Shouldn't the return value in the function signature be void as well? Moreover, here is the function exit: function exit%(%): int %{ exit(0); return 0; %} Shouldn't it return 'void' at the scripting layer since it shuts down the Bro process immediately? Why not let users control the return code like this: function exit%(code: int): void %{ exit(code); %} There seems to be another redundancy: function active_connection%(id: conn_id%): bool %{ Connection* c = sessions->FindConnection(id); return new Val(c ? 1 : 0, TYPE_BOOL); %} and function connection_exists%(c: conn_id%): bool %{ if ( sessions->FindConnection(c) ) return new Val(1, TYPE_BOOL); else return new Val(0, TYPE_BOOL); %} are essentially the same. Only the latter is used in the current set of scripts. Should we get rid of the former? The same sort of redundancy exists for: function connection_record%(cid: conn_id%): connection %{ Connection* c = sessions->FindConnection(cid); if ( c ) return c->BuildConnVal(); else { // Hard to recover from this until we have union types ... builtin_error("connection ID not a known connection (fatal)", cid); exit(0); return 0; } %} and function lookup_connection%(cid: conn_id%): connection %{ Connection* conn = sessions->FindConnection(cid); if ( conn ) return conn->BuildConnVal(); builtin_error("connection ID not a known connection", cid); // Return a dummy connection record. RecordVal* c = new RecordVal(connection_type); RecordVal* id_val = new RecordVal(conn_id); id_val->Assign(0, new AddrVal((unsigned int) 0)); id_val->Assign(1, new PortVal(ntohs(0), TRANSPORT_UDP)); id_val->Assign(2, new AddrVal((unsigned int) 0)); id_val->Assign(3, new PortVal(ntohs(0), TRANSPORT_UDP)); c->Assign(0, id_val); RecordVal* orig_endp = new RecordVal(endpoint); orig_endp->Assign(0, new Val(0, TYPE_COUNT)); orig_endp->Assign(1, new Val(int(0), TYPE_COUNT)); RecordVal* resp_endp = new RecordVal(endpoint); resp_endp->Assign(0, new Val(0, TYPE_COUNT)); resp_endp->Assign(1, new Val(int(0), TYPE_COUNT)); c->Assign(1, orig_endp); c->Assign(2, resp_endp); c->Assign(3, new Val(network_time, TYPE_TIME)); c->Assign(4, new Val(0.0, TYPE_INTERVAL)); c->Assign(5, new TableVal(string_set)); // service c->Assign(6, new StringVal("")); // addl c->Assign(7, new Val(0, TYPE_COUNT)); // hot c->Assign(8, new StringVal("")); // history return c; %} except that the latter returns a dummy connection value. Neither function is currently used at script land. My vote would be to remove the former one, as the latter has more graceful semantics. Matthias From gregor at icir.org Fri Nov 18 21:11:33 2011 From: gregor at icir.org (Gregor Maier) Date: Fri, 18 Nov 2011 21:11:33 -0800 Subject: [Bro-Dev] Some bro.bif issues In-Reply-To: <1321665790-sup-8037@samurai.local> References: <1321665790-sup-8037@samurai.local> Message-ID: <4EC73A85.6030808@icir.org> On 11/18/11 19:14 , Matthias Vallentin wrote: > This is the function set_buf: > > function set_buf%(f: file, buffered: bool%): any > %{ > f->SetBuf(buffered); > return new Val(0, TYPE_VOID); > %} > > It's return value is any although void is returned. Shouldn't the return value > in the function signature be void as well? Except for this one, all BiF's that don't have a return value are declared as returning "any" and will unconditionally do a "return 0"... AFAIK BiF's currently can't return void. Although I think that's just (an easily fixed) limitation of the BiF-parser. So I guess that either this BiF should be changed to return 0 to be consistent with other BiFs. Or the BiF-parser should be changed to allow void as return value and the BiF's updated accordingly. Might want to schedule this for 2.1 or 2.2.... > Moreover, here is the function exit: > > function exit%(%): int > %{ > exit(0); > return 0; > %} Per the above. Currently it should return an "any" to be consistent with other code. (Search for "return 0" in any BiF file and you'll find plenty of functions that should return void but do in fact return "any") > function active_connection%(id: conn_id%): bool VS. > function connection_exists%(c: conn_id%): bool > > are essentially the same. Only the latter is used in the current set of > scripts. Should we get rid of the former? sounds good. The latter is also the more intuitive name.... > The same sort of redundancy exists for: > > function connection_record%(cid: conn_id%): connection > function lookup_connection%(cid: conn_id%): connection > > except that the latter returns a dummy connection value. Neither function is > currently used at script land. My vote would be to remove the former one, as > the latter has more graceful semantics. ACK. We for sure shouldn't call exit() when the connection doesn't exist... cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From robin at icir.org Sat Nov 19 08:41:59 2011 From: robin at icir.org (Robin Sommer) Date: Sat, 19 Nov 2011 08:41:59 -0800 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/jsiwek/broxygen-cleanup: Teach Broxygen the .. bro:see:: directive (3056216) In-Reply-To: <201111191321.pAJDLFhY022431@bro-ids.icir.org> References: <201111191321.pAJDLFhY022431@bro-ids.icir.org> Message-ID: <20111119164159.GA80789@icir.org> > Teach Broxygen the .. bro:see:: directive Awesome, thanks! Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Sat Nov 19 15:05:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 19 Nov 2011 23:05:13 -0000 Subject: [Bro-Dev] #687: Smaller make trouble In-Reply-To: <047.35a104e568a47473cb847c4603f4a3eb@tracker.bro-ids.org> References: <047.35a104e568a47473cb847c4603f4a3eb@tracker.bro-ids.org> Message-ID: <062.c021e560ed5fabd406fa8defedde0bf7@tracker.bro-ids.org> #687: Smaller make trouble ----------------------------+-------------------- Reporter: robin | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: fixed | Keywords: ----------------------------+-------------------- Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [15eadb90b59de32af392dc33a80c3b589c303c3b/bro]: {{{ #!CommitTicketReference repository="bro" revision="15eadb90b59de32af392dc33a80c3b589c303c3b" Merge remote-tracking branch 'origin/topic/jsiwek/parallel-make-recursion' * origin/topic/jsiwek/parallel-make-recursion: Fix parallel make portability Revert "Fix parallel make from top-level to work on more platforms" Closes #687. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Nov 19 15:05:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 19 Nov 2011 23:05:13 -0000 Subject: [Bro-Dev] #650: Finish Broxygen docs In-Reply-To: <047.9abf8a0851b3b85991457973560c772f@tracker.bro-ids.org> References: <047.9abf8a0851b3b85991457973560c772f@tracker.bro-ids.org> Message-ID: <062.e67bdfd02a3aae3de9012f44a1b46821@tracker.bro-ids.org> #650: Finish Broxygen docs ----------------------------+-------------------- Reporter: robin | Owner: jsiwek Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: fixed | Keywords: ----------------------------+-------------------- Changes (by robin): * status: reopened => closed * resolution: => fixed Comment: In [b4cadd8f26d787cea4032a20edb2ab5b8ed1f4d1/bro]: {{{ #!CommitTicketReference repository="bro" revision="b4cadd8f26d787cea4032a20edb2ab5b8ed1f4d1" Merge remote-tracking branch 'origin/topic/jsiwek/broxygen-cleanup' * origin/topic/jsiwek/broxygen-cleanup: Teach Broxygen the .. bro:see:: directive More Broxygen stylesheet tweaks. Change external Broxygen links to render same as internals. Teach Broxygen to generate an index of Bro notices Teach Broxygen :bro:see: role for referencing any identifier in Bro domain. Add a "last updated date" to Broxygen docs. Closes #650. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Nov 19 16:25:19 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 20 Nov 2011 00:25:19 -0000 Subject: [Bro-Dev] #518: NMI: Use consistent version of CMake In-Reply-To: <050.2fd3ee8f00f4300b2ccd6054beddb68b@tracker.bro-ids.org> References: <050.2fd3ee8f00f4300b2ccd6054beddb68b@tracker.bro-ids.org> Message-ID: <065.476bd8b49dc898aacfa2ebc31bbc3717@tracker.bro-ids.org> #518: NMI: Use consistent version of CMake -----------------------+----------------------------- Reporter: appleman | Owner: jsiwek Type: Task | Status: closed Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: Rejected | Keywords: nmi build cmake -----------------------+----------------------------- Changes (by jsiwek): * status: assigned => closed * resolution: => Rejected Comment: This isn't needed anymore since we decided not to use NMI for building binaries. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Nov 20 08:05:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 20 Nov 2011 16:05:50 -0000 Subject: [Bro-Dev] #689: Adding CPU Profiler to FindGooglePerftools Message-ID: <048.77123b2850aa4bfb2ea8301ccf8d27a3@tracker.bro-ids.org> #689: Adding CPU Profiler to FindGooglePerftools --------------------+----------------------------- Reporter: jswaro | Type: Feature Request Status: new | Priority: Normal Milestone: | Component: Bro Version: | Keywords: --------------------+----------------------------- Currently, the CPU profiler is not included when --enable-perftools is enabled. The attached patch file from Gilbert Clark should fix this. -- Ticket URL: Bro Tracker Bro Issue Tracker From vallentin at icir.org Sun Nov 20 17:57:44 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Sun, 20 Nov 2011 17:57:44 -0800 Subject: [Bro-Dev] Status of anonymization framework Message-ID: <1321840483-sup-2064@samurai.local> I am about to document the following functions: preserve_prefix(a: addr, width: count): any preserve_subnet(sn: subnet): any anonymize_addr(a: addr, cl: IPAddrAnonymizationClass): addr I tried to play with them a bit but they don't seem to work out of the box. When I run preserve_prefix(1.2.3.4, 8); print anonymize_addr(1.2.3.4, ORIG_ADDR); print anonymize_addr(1.2.3.4, RESP_ADDR); print anonymize_addr(1.2.3.4, OTHER_ADDR); I get the following output: 1.2.3.4/8 1.2.3.4 1.2.3.4 1.2.3.4 Am I misusing the functions or have they been lost behind in the process of getting ready for 2.0? Matthias From vallentin at icir.org Sun Nov 20 18:00:05 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Sun, 20 Nov 2011 18:00:05 -0800 Subject: [Bro-Dev] Some bro.bif issues In-Reply-To: <4EC73A85.6030808@icir.org> References: <1321665790-sup-8037@samurai.local> <4EC73A85.6030808@icir.org> Message-ID: <1321840716-sup-8057@samurai.local> On Nov 18 2011 at 21:11PM PST, Gregor Maier wrote: > > function active_connection%(id: conn_id%): bool > > function connection_exists%(c: conn_id%): bool > > > > are essentially the same. Only the latter is used in the current set of > > scripts. Should we get rid of the former? > > sounds good. The latter is also the more intuitive name.... > > > The same sort of redundancy exists for: > > > > function connection_record%(cid: conn_id%): connection > > function lookup_connection%(cid: conn_id%): connection > > > > except that the latter returns a dummy connection value. Neither function is > > currently used at script land. My vote would be to remove the former one, as > > the latter has more graceful semantics. > > ACK. > We for sure shouldn't call exit() when the connection doesn't exist... Do others have any objections? Otherwise I would go ahead an remove them. Matthias From vallentin at icir.org Sun Nov 20 18:09:37 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Sun, 20 Nov 2011 18:09:37 -0800 Subject: [Bro-Dev] md5_hash vs. md5_hmac Message-ID: <1321841174-sup-7477@samurai.local> The implementation of md5_hash(...) and md5_hmac(...) in bro.bif is identical: %{ unsigned char digest[16]; hmac_md5_val(@ARG@, digest); return new StringVal(md5_digest_print(digest)); %} It seems that md5_hmac should support providing the secret key. Does anyone know more details about this? Matthias From vern at icir.org Sun Nov 20 18:23:59 2011 From: vern at icir.org (Vern Paxson) Date: Sun, 20 Nov 2011 18:23:59 -0800 Subject: [Bro-Dev] Status of anonymization framework In-Reply-To: <1321840483-sup-2064@samurai.local> (Sun, 20 Nov 2011 17:57:44 PST). Message-ID: <20111121022359.482F82C4002@rock.ICSI.Berkeley.EDU> > I am about to document the following functions: I believe this are for use with -A (anonymize/transform tracefile). Vern From vallentin at icir.org Sun Nov 20 18:30:15 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Sun, 20 Nov 2011 18:30:15 -0800 Subject: [Bro-Dev] Status of anonymization framework In-Reply-To: <20111121022359.482F82C4002@rock.ICSI.Berkeley.EDU> References: <20111121022359.482F82C4002@rock.ICSI.Berkeley.EDU> Message-ID: <1321842528-sup-3997@samurai.local> > I believe this are for use with -A (anonymize/transform tracefile). In git/master, this option does not exist (anymore). Matthias From bro at tracker.bro-ids.org Sun Nov 20 21:46:55 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 21 Nov 2011 05:46:55 -0000 Subject: [Bro-Dev] #684: Change to_port BiF to take a string as argument In-Reply-To: <050.b20757078e176ef09b16f1693c6f73e6@tracker.bro-ids.org> References: <050.b20757078e176ef09b16f1693c6f73e6@tracker.bro-ids.org> Message-ID: <065.492833e603287df727983fb8facb9372@tracker.bro-ids.org> #684: Change to_port BiF to take a string as argument -----------------------+------------------------ Reporter: matthias | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.1 Component: Bro | Version: git/master Resolution: | Keywords: -----------------------+------------------------ Comment (by matthias): In [0325b5ea32f9b86c89ca020097de4586d782f636/bro]: {{{ #!CommitTicketReference repository="bro" revision="0325b5ea32f9b86c89ca020097de4586d782f636" to_port() now parses a string instead of a count. Addresses #684. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Nov 20 21:51:23 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 21 Nov 2011 05:51:23 -0000 Subject: [Bro-Dev] #684: Change to_port BiF to take a string as argument In-Reply-To: <050.b20757078e176ef09b16f1693c6f73e6@tracker.bro-ids.org> References: <050.b20757078e176ef09b16f1693c6f73e6@tracker.bro-ids.org> Message-ID: <065.861712f724a303473317899eac872fe2@tracker.bro-ids.org> #684: Change to_port BiF to take a string as argument -----------------------+------------------------ Reporter: matthias | Owner: Type: Problem | Status: testing Priority: Normal | Milestone: Bro2.1 Component: Bro | Version: git/master Resolution: | Keywords: -----------------------+------------------------ Changes (by matthias): * status: new => testing -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Nov 20 21:55:51 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 21 Nov 2011 05:55:51 -0000 Subject: [Bro-Dev] #684: Change to_port BiF to take a string as argument In-Reply-To: <050.b20757078e176ef09b16f1693c6f73e6@tracker.bro-ids.org> References: <050.b20757078e176ef09b16f1693c6f73e6@tracker.bro-ids.org> Message-ID: <065.e45f3a33fca53a7ec365225f014c719d@tracker.bro-ids.org> #684: Change to_port BiF to take a string as argument ----------------------------+------------------------ Reporter: matthias | Owner: matthias Type: Merge Request | Status: accepted Priority: Normal | Milestone: Bro2.1 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by matthias): * owner: => matthias * status: testing => accepted * type: Problem => Merge Request -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Mon Nov 21 08:15:52 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 21 Nov 2011 08:15:52 -0800 Subject: [Bro-Dev] Status of anonymization framework In-Reply-To: <1321842528-sup-3997@samurai.local> References: <20111121022359.482F82C4002@rock.ICSI.Berkeley.EDU> <1321842528-sup-3997@samurai.local> Message-ID: <20111121161552.GD34031@icir.org> On Sun, Nov 20, 2011 at 18:30 -0800, you wrote: > In git/master, this option does not exist (anymore). We removed the rewriting code. But do the bifs depend on that code, or should they work independently as well ? If they do depend on it, it we should remove them as well (or fix them). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Mon Nov 21 08:18:25 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 21 Nov 2011 08:18:25 -0800 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/bernhard/input: first test. (029871e) In-Reply-To: <201111202142.pAKLgR2F031436@bro-ids.icir.org> References: <201111202142.pAKLgR2F031436@bro-ids.icir.org> Message-ID: <20111121161825.GA34249@icir.org> On Sun, Nov 20, 2011 at 13:42 -0800, Bernhard wrote: > + # first read in the old stuff into the table... > + Input::create_stream(A::LOG, [$source="input.log"]); > + Input::add_filter(A::LOG, [$name="ssh", $idx=idx, $val=val, $destination=servers]); > + Input::force_update(A::LOG); > + print servers; Nice! Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Mon Nov 21 08:27:26 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 21 Nov 2011 08:27:26 -0800 Subject: [Bro-Dev] Some bro.bif issues In-Reply-To: <1321840716-sup-8057@samurai.local> References: <1321665790-sup-8037@samurai.local> <4EC73A85.6030808@icir.org> <1321840716-sup-8057@samurai.local> Message-ID: <20111121162725.GE34249@icir.org> On Sun, Nov 20, 2011 at 18:00 -0800, you wrote: > Do others have any objections? Otherwise I would go ahead an remove > them. Cleaning these things up makes sense. Just commit these all to the cleanup branch you created, and we can merge that in when 2.1 devel starts. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Mon Nov 21 09:10:16 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 21 Nov 2011 17:10:16 -0000 Subject: [Bro-Dev] #689: Adding CPU Profiler to FindGooglePerftools In-Reply-To: <048.77123b2850aa4bfb2ea8301ccf8d27a3@tracker.bro-ids.org> References: <048.77123b2850aa4bfb2ea8301ccf8d27a3@tracker.bro-ids.org> Message-ID: <063.3be64cf206e6791c4cab3adafa7a849b@tracker.bro-ids.org> #689: Adding CPU Profiler to FindGooglePerftools ---------------------+-------------------- Reporter: jswaro | Owner: Type: Patch | Status: new Priority: Normal | Milestone: Bro2.1 Component: Bro | Version: Resolution: | Keywords: ---------------------+-------------------- Changes (by robin): * type: Feature Request => Patch * milestone: => Bro2.1 -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Nov 21 09:17:00 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 21 Nov 2011 17:17:00 -0000 Subject: [Bro-Dev] #690: GTP de-tunneling Message-ID: <047.bf6216f5c5ffabe44fd63ab9aa939d04@tracker.bro-ids.org> #690: GTP de-tunneling ------------------------+--------------------- Reporter: robin | Type: Problem Status: new | Priority: Normal Milestone: 2.1 | Component: Bro Version: git/master | ------------------------+--------------------- ----- Forwarded message from "Langer, Carsten (NSN - DE/Duesseldorf)" ----- From: "Langer, Carsten (NSN - DE/Duesseldorf)" [...] I lately developed a GTP (GPRS Tunneling Protocol) de-tunneling functionality for bro-ids, which I want to share with you in the hope that you might find it helpful. [...] Please find attached a patched version of the Sessions.cc, where from line 601 to 701 I have introduced the de-GTP stuff. This is based on bro-ids v1.5.3. I found that patching this one single location was good enough for my purpose. It works for me, however I could only test it against a couple of network traces that I have, so if you are interested to re-use the patch, please give it a try against other sets of data as well. I added 3 weird-warnings (lines 614, 674, 691) if something goes wrong within the patch, but haven't updated anything in the weird.bro script. [...] As I'm not using the bro-ids for network security analysis but for application performance analysis, [...] I did not take any measures against recursive GTP tunnels. ----- End forwarded message ----- [attachment:"Sessions_patched_for_gtp-detunneling.cc"] -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Mon Nov 21 09:17:57 2011 From: seth at icir.org (Seth Hall) Date: Mon, 21 Nov 2011 12:17:57 -0500 Subject: [Bro-Dev] Some bro.bif issues In-Reply-To: <1321840716-sup-8057@samurai.local> References: <1321665790-sup-8037@samurai.local> <4EC73A85.6030808@icir.org> <1321840716-sup-8057@samurai.local> Message-ID: On Nov 20, 2011, at 9:00 PM, Matthias Vallentin wrote: > Do others have any objections? Otherwise I would go ahead an remove > them. All of this is welcome change! The duplication and inconsistency with some of these functions has driven me crazy for a long time. Thanks, .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Mon Nov 21 09:18:47 2011 From: seth at icir.org (Seth Hall) Date: Mon, 21 Nov 2011 12:18:47 -0500 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/bernhard/input: first test. (029871e) In-Reply-To: <20111121161825.GA34249@icir.org> References: <201111202142.pAKLgR2F031436@bro-ids.icir.org> <20111121161825.GA34249@icir.org> Message-ID: <9C9034E2-1FAD-486D-82E2-6354D5B1AE9E@icir.org> On Nov 21, 2011, at 11:18 AM, Robin Sommer wrote: > On Sun, Nov 20, 2011 at 13:42 -0800, Bernhard wrote: > >> + # first read in the old stuff into the table... >> + Input::create_stream(A::LOG, [$source="input.log"]); >> + Input::add_filter(A::LOG, [$name="ssh", $idx=idx, $val=val, $destination=servers]); >> + Input::force_update(A::LOG); >> + print servers; > > Nice! Seriously! This is awesome! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Mon Nov 21 09:20:45 2011 From: seth at icir.org (Seth Hall) Date: Mon, 21 Nov 2011 12:20:45 -0500 Subject: [Bro-Dev] Status of anonymization framework In-Reply-To: <20111121161552.GD34031@icir.org> References: <20111121022359.482F82C4002@rock.ICSI.Berkeley.EDU> <1321842528-sup-3997@samurai.local> <20111121161552.GD34031@icir.org> Message-ID: <5341503C-AF98-4E7B-8761-E42F6B7639BE@icir.org> On Nov 21, 2011, at 11:15 AM, Robin Sommer wrote: > On Sun, Nov 20, 2011 at 18:30 -0800, you wrote: > >> In git/master, this option does not exist (anymore). > > We removed the rewriting code. But do the bifs depend on that code, or > should they work independently as well ? If they do depend on it, it > we should remove them as well (or fix them). I think it makes sense to maintain anonymization primitives like this which we can then reuse in the logging framework for log anonymization when we get to that. At the moment though I don't think they serve much purpose especially if they don't work or they don't work like you would anticipate. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From vern at icir.org Mon Nov 21 10:26:15 2011 From: vern at icir.org (Vern Paxson) Date: Mon, 21 Nov 2011 10:26:15 -0800 Subject: [Bro-Dev] md5_hash vs. md5_hmac In-Reply-To: <1321841174-sup-7477@samurai.local> (Sun, 20 Nov 2011 18:09:37 PST). Message-ID: <20111121182615.A63EE2C4012@rock.ICSI.Berkeley.EDU> > The implementation of md5_hash(...) and md5_hmac(...) in bro.bif is > identical: > > %{ > unsigned char digest[16]; > hmac_md5_val(@ARG@, digest); > return new StringVal(md5_digest_print(digest)); > %} They're not identical. (At least, not in 1.5, which is what I can easily check.) md5_hash() calls hash_md5_val() rather than hmac_md5_val(). Per the CHANGES: - The new built-in md5_hmac() returns an HMAC-MD5 hash of the given string (Ruoming Pang). The HMAC secret key is generated from available entropy when Bro starts up, or it can be specified for repeatability using the new -K flag. Vern From seth at icir.org Mon Nov 21 10:30:50 2011 From: seth at icir.org (Seth Hall) Date: Mon, 21 Nov 2011 13:30:50 -0500 Subject: [Bro-Dev] md5_hash vs. md5_hmac In-Reply-To: <20111121182615.A63EE2C4012@rock.ICSI.Berkeley.EDU> References: <20111121182615.A63EE2C4012@rock.ICSI.Berkeley.EDU> Message-ID: On Nov 21, 2011, at 1:26 PM, Vern Paxson wrote: > They're not identical. (At least, not in 1.5, which is what I can easily > check.) md5_hash() calls hash_md5_val() rather than hmac_md5_val(). > Per the CHANGES: > > - The new built-in md5_hmac() returns an HMAC-MD5 hash of the given string > (Ruoming Pang). The HMAC secret key is generated from available entropy > when Bro starts up, or it can be specified for repeatability using > the new -K flag. That should still be correct since I wasn't looking for an HMAC value with the md5_hash function and we haven't touched that code since it was added. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Mon Nov 21 10:57:20 2011 From: seth at icir.org (Seth Hall) Date: Mon, 21 Nov 2011 13:57:20 -0500 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/bernhard/input: first test. (029871e) In-Reply-To: <201111202142.pAKLgR2F031436@bro-ids.icir.org> References: <201111202142.pAKLgR2F031436@bro-ids.icir.org> Message-ID: <30F1A87B-5888-4991-A980-60EAFD237C6B@icir.org> On Nov 20, 2011, at 4:42 PM, Bernhard Amann wrote: > +type idx: record { > > +type val: record { I know this is super early code, but could make this follow the same style as the rest of the scripts? (camel-casing for types) http://www.bro-ids.org/development/script-conventions.html > + Input::add_filter(A::LOG, [$name="ssh", $idx=idx, $val=val, $destination=servers]); I also have a question about the add_filter line. I see that the "i" field is used for the index and everything else is used for the val. Could those fields be mixed? The "i" field used in the val and fields from the val used in the index? I guess I'm really just asking if I could use a field in both the index and the value. Thanks! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From vallentin at icir.org Mon Nov 21 11:01:01 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Mon, 21 Nov 2011 11:01:01 -0800 Subject: [Bro-Dev] md5_hash vs. md5_hmac In-Reply-To: References: <20111121182615.A63EE2C4012@rock.ICSI.Berkeley.EDU> Message-ID: <1321901914-sup-9583@samurai.local> > That should still be correct since I wasn't looking for an HMAC value > with the md5_hash function and we haven't touched that code since it > was added. I had to squint my eyes intensively to see the difference in the function bodies: hash_md5_val(@ARG@, digest); vs. hmac_md5_val(@ARG@, digest); That makes more sense now to me, sorry for the noise. Matthias From bro at tracker.bro-ids.org Mon Nov 21 17:27:03 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 22 Nov 2011 01:27:03 -0000 Subject: [Bro-Dev] #691: Clean up login/telnet/rlogin events Message-ID: <047.d6b4034016b922b40f9a1ded28fac7a0@tracker.bro-ids.org> #691: Clean up login/telnet/rlogin events ---------------------+----------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Component: Bro | Version: Keywords: cleanup | ---------------------+----------------- This is old stuff but still worth cleaning up: naming, semantics, and arguments of the various rlogin/rsh/telnet events is a mess; see event `login_*`, `authentication_*`, and `rsh_*`. This needs to be unified. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Nov 22 09:07:57 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 22 Nov 2011 17:07:57 -0000 Subject: [Bro-Dev] #695: ssl_extension event doesn't indicate originator Message-ID: <046.1ac51e555491fc9166b30af08cbf08ab@tracker.bro-ids.org> #695: ssl_extension event doesn't indicate originator ---------------------+-------------------- Reporter: seth | Owner: seth Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Keywords: | ---------------------+-------------------- I forgot to include which side of the connection sent an ssl_extension. The ssl_alert, x509_extension, and x509_error events need similar fields added as well. The fields should probably be named consistently with the other analyzer generated events as well (is_orig). The x509_certificate event uses inconsistent naming (is_server) and should probably be renamed too. -- Ticket URL: Bro Tracker Bro Issue Tracker From hlin33 at illinois.edu Tue Nov 22 09:37:37 2011 From: hlin33 at illinois.edu (Hui Lin (Hugo) ) Date: Tue, 22 Nov 2011 11:37:37 -0600 Subject: [Bro-Dev] Hui Lin_Searching help for some reconstructing TCP packet Message-ID: Hi, It is probably not related to Bro development but it is a challenge that I met in my research progress. Right now, I can at least get some different DNP3 traffic from what I had before. I would like to generate some illegal traffic based on this legal traffic set. My plan is to flip a bit values in each bit location of the DNP3 packet and see how my Bro analyzer behave. For each DNP3 packet, I can use the Bro to extract a byte stream of the TCP payload. I am wondering is there any way to reverse this procedure. For example, I had a byte stream X. I wish to construct a network packet based on TCP protocol and this X will be the payload of TCP packet. If the X is too long, I hope that several TCP packets can be constructed with the correct sequence number being set. I am taking look at the socket api in Windows at this moment (WSASocket). But I am not quite sure whether it is possible to do this by that API. Any comments and inputs are welcome. Best -- Hui Lin Research Assistant DEPEND Research Group, ECE Department University of Illinois at Urbana-Champaign hlin33 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20111122/3aeefa08/attachment.html From seth at icir.org Tue Nov 22 09:40:32 2011 From: seth at icir.org (Seth Hall) Date: Tue, 22 Nov 2011 12:40:32 -0500 Subject: [Bro-Dev] Hui Lin_Searching help for some reconstructing TCP packet In-Reply-To: References: Message-ID: <791282C8-5018-421F-BDAB-AD27B91C7A82@icir.org> On Nov 22, 2011, at 12:37 PM, Hui Lin (Hugo) wrote: > For each DNP3 packet, I can use the Bro to extract a byte stream of the TCP payload. I am wondering is there any way to reverse this procedure. For example, I had a byte stream X. I wish to construct a network packet based on TCP protocol and this X will be the payload of TCP packet. If the X is too long, I hope that several TCP packets can be constructed with the correct sequence number being set. It might be best to use something like Scapy to modify the packet contents. I think you should be able to load in packets, modify them and write them back out. (don't hold me to that though, I don't know Scapy very well yet) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Tue Nov 22 14:43:17 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 22 Nov 2011 22:43:17 -0000 Subject: [Bro-Dev] #696: broctl dumps python traceback information when bro is communicating with other worker nodes and quits Message-ID: <049.98d6f78884b685b89df7ef541cd92394@tracker.bro-ids.org> #696: broctl dumps python traceback information when bro is communicating with other worker nodes and quits ---------------------+--------------------- Reporter: aashish | Type: Problem Status: new | Priority: Normal Milestone: | Component: Bro Version: 1.5.3 | Keywords: ---------------------+--------------------- broctl dumps python traceback information when bro is communicating (ssh) with other worker nodes and this continues until the ssh connection is over. $ broctl check Traceback (most recent call last): File "/home/users/bro/cluster/bin/broctl", line 726, in loop.onecmd(line) File "/usr/local/lib/python2.6/cmd.py", line 219, in onecmd return func(arg) File "/home/users/bro/cluster/bin/broctl", line 359, in do_check self.lock() File "/home/users/bro/cluster/bin/broctl", line 96, in lock if not util.lock(): File "/home/users/bro/cluster/lib/broctl/BroControl/util.py", line 180, in lock if do_ouput: NameError: global name 'do_ouput' is not defined Process info as follows: [bro at host]$ ps auxwww | fgrep bro bro 12704 1.6 0.2 10208 6676 ?? D 2:30PM 0:01.85 python /home/users/bro/cluster/bin/broctl cron bro 12613 0.0 0.1 4464 1716 ?? Is 2:30PM 0:00.01 /bin/csh -c /home/users/bro/cluster/bin/broctl cron bro 12743 0.0 0.0 3464 1120 ?? Is 2:30PM 0:00.00 /bin/sh -c sh bro 12745 0.0 0.0 3464 1120 ?? I 2:30PM 0:00.00 sh bro 12755 0.0 0.0 3464 1120 ?? Is 2:30PM 0:00.01 /bin/sh -c ssh -o ConnectTimeout=30 -l bro bc2-proxy sh bro 12757 0.0 0.1 5560 2644 ?? I 2:30PM 0:00.03 ssh -o ConnectTimeout=30 -l bro bc2-proxy sh bro 12766 0.0 0.0 3464 1120 ?? Is 2:30PM 0:00.00 /bin/sh -c ssh -o ConnectTimeout=30 -l bro bc2-01 sh bro 12768 0.0 0.1 5560 2644 ?? I 2:30PM 0:00.03 ssh -o ConnectTimeout=30 -l bro bc2-01 sh bro 12772 0.0 0.0 3464 1120 ?? Is 2:30PM 0:00.00 /bin/sh -c ssh -o ConnectTimeout=30 -l bro bc2-10 sh bro 12774 0.0 0.1 5560 2644 ?? I 2:30PM 0:00.03 ssh -o ConnectTimeout=30 -l bro bc2-10 sh bro 12779 0.0 0.0 3464 1120 ?? Is 2:30PM 0:00.00 /bin/sh -c ssh -o ConnectTimeout=30 -l bro bc2-02 sh bro 12781 0.0 0.1 5560 2644 ?? I 2:30PM 0:00.03 ssh -o ConnectTimeout=30 -l bro bc2-02 sh bro 12793 0.0 0.0 3464 1120 ?? Is 2:30PM 0:00.00 /bin/sh -c ssh -o ConnectTimeout=30 -l bro bc2-03 sh bro 12795 0.0 0.1 5560 2644 ?? I 2:30PM 0:00.03 ssh -o ConnectTimeout=30 -l bro bc2-03 sh bro 12798 0.0 0.0 3464 1120 ?? Is 2:30PM 0:00.00 /bin/sh -c ssh -o ConnectTimeout=30 -l bro bc2-04 sh bro 12800 0.0 0.1 5560 2644 ?? I 2:30PM 0:00.03 ssh -o ConnectTimeout=30 -l bro bc2-04 sh bro 12803 0.0 0.0 3464 1120 ?? Is 2:30PM 0:00.00 /bin/sh -c ssh -o ConnectTimeout=30 -l bro bc2-05 sh bro 12805 0.0 0.1 5560 2644 ?? I 2:30PM 0:00.04 ssh -o ConnectTimeout=30 -l bro bc2-05 sh bro 12815 0.0 0.0 3464 1120 ?? Is 2:30PM 0:00.00 /bin/sh -c ssh -o ConnectTimeout=30 -l bro bc2-06 sh bro 12817 0.0 0.1 5560 2644 ?? I 2:30PM 0:00.04 ssh -o ConnectTimeout=30 -l bro bc2-06 sh bro 12820 0.0 0.0 3464 1120 ?? Is 2:30PM 0:00.00 /bin/sh -c ssh -o ConnectTimeout=30 -l bro bc2-07 sh bro 12822 0.0 0.1 5560 2644 ?? I 2:30PM 0:00.03 ssh -o ConnectTimeout=30 -l bro bc2-07 sh bro 12825 0.0 0.0 3464 1120 ?? Is 2:30PM 0:00.00 /bin/sh -c ssh -o ConnectTimeout=30 -l bro bc2-08 sh bro 12827 0.0 0.1 5560 2644 ?? I 2:30PM 0:00.03 ssh -o ConnectTimeout=30 -l bro bc2-08 sh bro 12830 0.0 0.0 3464 1120 ?? Is 2:30PM 0:00.00 /bin/sh -c ssh -o ConnectTimeout=30 -l bro bc2-09 sh bro 12831 0.0 0.1 5560 2644 ?? I 2:30PM 0:00.03 ssh -o ConnectTimeout=30 -l bro bc2-09 sh -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Tue Nov 22 14:45:43 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 22 Nov 2011 14:45:43 -0800 Subject: [Bro-Dev] #696: broctl dumps python traceback information when bro is communicating with other worker nodes and quits In-Reply-To: <049.98d6f78884b685b89df7ef541cd92394@tracker.bro-ids.org> References: <049.98d6f78884b685b89df7ef541cd92394@tracker.bro-ids.org> Message-ID: <20111122224543.GD57617@icir.org> On Tue, Nov 22, 2011 at 22:43 -0000, you wrote: > File "/home/users/bro/cluster/lib/broctl/BroControl/util.py", line 180, > in lock > if do_ouput: > NameError: global name 'do_ouput' is not defined Ah, that one is known and fixed in 2.0: just edit the line and fix the spelling to "do_output". -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Tue Nov 22 14:45:45 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 22 Nov 2011 22:45:45 -0000 Subject: [Bro-Dev] #696: broctl dumps python traceback information when bro is communicating with other worker nodes and quits In-Reply-To: <049.98d6f78884b685b89df7ef541cd92394@tracker.bro-ids.org> References: <049.98d6f78884b685b89df7ef541cd92394@tracker.bro-ids.org> Message-ID: <064.8345e27be7c3743c8c51bae2cb8b675e@tracker.bro-ids.org> #696: broctl dumps python traceback information when bro is communicating with other worker nodes and quits ----------------------+------------------- Reporter: aashish | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: 1.5.3 Resolution: | Keywords: ----------------------+------------------- Comment (by robin): On Tue, Nov 22, 2011 at 22:43 -0000, you wrote: > File "/home/users/bro/cluster/lib/broctl/BroControl/util.py", line 180, > in lock > if do_ouput: > NameError: global name 'do_ouput' is not defined Ah, that one is known and fixed in 2.0: just edit the line and fix the spelling to "do_output". -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 23 08:35:57 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 23 Nov 2011 16:35:57 -0000 Subject: [Bro-Dev] #695: ssl_extension event doesn't indicate originator In-Reply-To: <046.1ac51e555491fc9166b30af08cbf08ab@tracker.bro-ids.org> References: <046.1ac51e555491fc9166b30af08cbf08ab@tracker.bro-ids.org> Message-ID: <061.e9d328900db818bf06d613f13d723136@tracker.bro-ids.org> #695: ssl_extension event doesn't indicate originator ----------------------+-------------------- Reporter: seth | Owner: seth Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by robin): On Tue, Nov 22, 2011 at 17:07 -0000, you wrote: > I forgot to include which side of the connection sent an ssl_extension. > The ssl_alert, x509_extension, and x509_error events need similar fields > added as well. > > The fields should probably be named consistently with the other analyzer > generated events as well (is_orig). The x509_certificate event uses > inconsistent naming (is_server) and should probably be renamed too. I'm be fine fixing these for 2.0. Sounds easy enough to fix and worth doing now before people start relying on it. -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Wed Nov 23 10:06:40 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 23 Nov 2011 10:06:40 -0800 Subject: [Bro-Dev] Hui Lin_Searching help for some reconstructing TCP packet In-Reply-To: <791282C8-5018-421F-BDAB-AD27B91C7A82@icir.org> References: <791282C8-5018-421F-BDAB-AD27B91C7A82@icir.org> Message-ID: <20111123180640.GK89839@icir.org> On Tue, Nov 22, 2011 at 12:40 -0500, you wrote: > It might be best to use something like Scapy to modify the packet > contents. netdude may work as well: netdude.sourceforge.net/ Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Wed Nov 23 13:31:21 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 23 Nov 2011 21:31:21 -0000 Subject: [Bro-Dev] #697: Equivalent of capture-events.bro in 2.x Message-ID: <050.4a84d4a1f0ff3e14c2a28b6cd50f737e@tracker.bro-ids.org> #697: Equivalent of capture-events.bro in 2.x ----------------------+------------------------ Reporter: matthias | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.1 Component: Bro | Version: git/master Keywords: | ----------------------+------------------------ How should we handle the functionality provided by the 1.5 script `capture-events.bro` in 2.x? It currently does not exist. Since it's implementation only consists of this one-liner {{{ event bro_init() { capture_events("events.bst"); } }}} I think we make that a redefinable script variable rather than shipping a separate script. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 23 13:38:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 23 Nov 2011 21:38:05 -0000 Subject: [Bro-Dev] #697: Equivalent of capture-events.bro in 2.x In-Reply-To: <050.4a84d4a1f0ff3e14c2a28b6cd50f737e@tracker.bro-ids.org> References: <050.4a84d4a1f0ff3e14c2a28b6cd50f737e@tracker.bro-ids.org> Message-ID: <065.f33918c5ab2da247137ff5055195eacf@tracker.bro-ids.org> #697: Equivalent of capture-events.bro in 2.x -----------------------+------------------------ Reporter: matthias | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.1 Component: Bro | Version: git/master Resolution: | Keywords: -----------------------+------------------------ Comment (by seth): > I think we make that a redefinable script variable rather than shipping a > separate script. That makes sense. We would be able to dump all events at the command line then and choose which file name to put them into. {{{ bro -r traffic.trace capture_events=events.bst }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 23 13:40:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 23 Nov 2011 21:40:49 -0000 Subject: [Bro-Dev] #695: ssl_extension event doesn't indicate originator In-Reply-To: <046.1ac51e555491fc9166b30af08cbf08ab@tracker.bro-ids.org> References: <046.1ac51e555491fc9166b30af08cbf08ab@tracker.bro-ids.org> Message-ID: <061.d818b7b410c0bb6446645c66078af77d@tracker.bro-ids.org> #695: ssl_extension event doesn't indicate originator ------------------------+-------------------- Reporter: seth | Owner: seth Type: Problem | Status: closed Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: Duplicate | Keywords: ------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Duplicate -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 23 13:41:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 23 Nov 2011 21:41:07 -0000 Subject: [Bro-Dev] #694: ssl_extension event doesn't indicate originator In-Reply-To: <046.d65e75b61df72b94900f8d9f5a13cc23@tracker.bro-ids.org> References: <046.d65e75b61df72b94900f8d9f5a13cc23@tracker.bro-ids.org> Message-ID: <061.c21f4d3cac652ba347c294ece682ab47@tracker.bro-ids.org> #694: ssl_extension event doesn't indicate originator ------------------------+-------------------- Reporter: seth | Owner: seth Type: Problem | Status: closed Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: Duplicate | Keywords: ------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Duplicate -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 23 13:41:25 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 23 Nov 2011 21:41:25 -0000 Subject: [Bro-Dev] #693: ssl_extension event doesn't indicate originator In-Reply-To: <046.bdda3794b4d1b3ea6a0a064ff6c689cc@tracker.bro-ids.org> References: <046.bdda3794b4d1b3ea6a0a064ff6c689cc@tracker.bro-ids.org> Message-ID: <061.0b919212a48bd3e911f0fc78baff113c@tracker.bro-ids.org> #693: ssl_extension event doesn't indicate originator ------------------------+-------------------- Reporter: seth | Owner: seth Type: Problem | Status: closed Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: Duplicate | Keywords: ------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Duplicate -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 23 14:26:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 23 Nov 2011 22:26:50 -0000 Subject: [Bro-Dev] #697: Equivalent of capture-events.bro in 2.x In-Reply-To: <050.4a84d4a1f0ff3e14c2a28b6cd50f737e@tracker.bro-ids.org> References: <050.4a84d4a1f0ff3e14c2a28b6cd50f737e@tracker.bro-ids.org> Message-ID: <065.ba4db9c85feb766fedf392ba56a23d01@tracker.bro-ids.org> #697: Equivalent of capture-events.bro in 2.x -----------------------+------------------------ Reporter: matthias | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.1 Component: Bro | Version: git/master Resolution: | Keywords: -----------------------+------------------------ Comment (by matthias): Replying to [comment:0 seth]: > {{{ > bro -r traffic.trace capture_events=events.bst > }}} Now to be really consistent, the counterpart to `-x` could be `-X`, giving us {{{ bro -r traffic.trace -X events.bst }}} for recording and {{{ bro -r traffic.trace -x events.bst }}} for replay. -- Ticket URL: Bro Tracker Bro Issue Tracker From hlin33 at illinois.edu Sun Nov 27 18:38:31 2011 From: hlin33 at illinois.edu (Hui Lin (Hugo) ) Date: Sun, 27 Nov 2011 20:38:31 -0600 Subject: [Bro-Dev] Hui Lin_can I directly call linux shell script within event handler Message-ID: Hi, Within Bro's event handler, can I use bro's script to call Linux shell script or perl script directly? If not, I think at least, I can let bro to communicate to broccoli's client and let it to call a shell script. Best, Hui -- Hui Lin Research Assistant DEPEND Research Group, ECE Department University of Illinois at Urbana-Champaign hlin33 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20111127/f3320819/attachment.html From vallentin at icir.org Sun Nov 27 20:42:14 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Sun, 27 Nov 2011 20:42:14 -0800 Subject: [Bro-Dev] Hui Lin_can I directly call linux shell script within event handler In-Reply-To: References: Message-ID: <1322455331-sup-9894@samurai.local> > Within Bro's event handler, can I use bro's script to call Linux shell > script or perl script directly? You can use Bro's system function to execute an arbitrary command: system(s: string): int Invokes a command via the system function. Returns true if the return value of system was non-zero. Returns the return value from the system() call. Note that this corresponds to the status of backgrounding the given command, not to the exit status of the command itself. A value of 127 corresponds to a failure to execute sh, and -1 to an internal system failure. Furthermore, the command is run in the background with stdout redirected to stderr It is good idea to call str_shell_escape(source: string): string on the argument to system. Matthias From vallentin at icir.org Sun Nov 27 20:44:40 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Sun, 27 Nov 2011 20:44:40 -0800 Subject: [Bro-Dev] BiF prefix Message-ID: <1322455365-sup-5942@samurai.local> Why does the BiF NFS3::mode2string%(mode: count%): string have a prefix of NFS3? No other BiF has such a prefix. Matthias From vern at icir.org Sun Nov 27 20:47:25 2011 From: vern at icir.org (Vern Paxson) Date: Sun, 27 Nov 2011 20:47:25 -0800 Subject: [Bro-Dev] BiF prefix In-Reply-To: <1322455365-sup-5942@samurai.local> (Sun, 27 Nov 2011 20:44:40 PST). Message-ID: <20111128044725.86AE32C4002@rock.ICSI.Berkeley.EDU> > Why does the BiF > > NFS3::mode2string%(mode: count%): string > > have a prefix of NFS3? Short version: blame Gregor, no doubt :-). Slighty longer version: Presumably a "mode" in this context refers to an NFSv3 protocol value that isn't meaningful in other contexts. Vern From vallentin at icir.org Sun Nov 27 20:54:36 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Sun, 27 Nov 2011 20:54:36 -0800 Subject: [Bro-Dev] BiF prefix In-Reply-To: <20111128044725.86AE32C4002@rock.ICSI.Berkeley.EDU> References: <20111128044725.86AE32C4002@rock.ICSI.Berkeley.EDU> Message-ID: <1322455817-sup-8692@samurai.local> > Slighty longer version: Presumably a "mode" in this context refers to an > NFSv3 protocol value that isn't meaningful in other contexts. That's what I was thinking, but looking at the implementation suggests that it's quite a generic way of converting UNIX files permissions given by a mode into a string of the form rw[xsS]rw[xsS]rw[xtT]. If I'm not overseeing a detail, I would suggest removing the prefix for consistency and to indicate the broader usage scenario. Matthias From vern at icir.org Sun Nov 27 21:22:58 2011 From: vern at icir.org (Vern Paxson) Date: Sun, 27 Nov 2011 21:22:58 -0800 Subject: [Bro-Dev] BiF prefix In-Reply-To: <1322455817-sup-8692@samurai.local> (Sun, 27 Nov 2011 20:54:36 PST). Message-ID: <20111128052258.25FCD2C4002@rock.ICSI.Berkeley.EDU> > that it's quite a generic way of converting UNIX files permissions given > by a mode into a string of the form rw[xsS]rw[xsS]rw[xtT]. > > If I'm not overseeing a detail, I would suggest removing the prefix for > consistency and to indicate the broader usage scenario. And maybe change it to have "file_mode" or "Unix_file_mode" in the name. Vern From seth at icir.org Mon Nov 28 05:12:43 2011 From: seth at icir.org (Seth Hall) Date: Mon, 28 Nov 2011 08:12:43 -0500 Subject: [Bro-Dev] Hui Lin_can I directly call linux shell script within event handler In-Reply-To: <1322455331-sup-9894@samurai.local> References: <1322455331-sup-9894@samurai.local> Message-ID: <201C4CAE-6032-4B14-B50F-9AF8C4D41350@icir.org> On Nov 27, 2011, at 11:42 PM, Matthias Vallentin wrote: > It is good idea to call > > str_shell_escape(source: string): string > > on the argument to system. That function is actually much less necessary now that we have the piped_exec function. str_shell_escape is only supposed to be used for data being put between double quotes so that the input doesn't escape from the double quoting. This is obviously super dangerous: system(fmt("rm \"%s\"", sniffed_data)); This is better: system(fmt("rm \"%s\"", str_shell_escape(sniffed_data))); If you need to supply something on STDIN to the program you are executing, you should use the piped_exec function. You can provide the data supplied on STDIN as an argument to the function. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Mon Nov 28 12:25:17 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 28 Nov 2011 20:25:17 -0000 Subject: [Bro-Dev] #655: broctl 'scripts' command doesn't work in cluster mode In-Reply-To: <048.94750b9656841ef5d3bd2012524def39@tracker.bro-ids.org> References: <048.94750b9656841ef5d3bd2012524def39@tracker.bro-ids.org> Message-ID: <063.4965c9e63ba6e8904d0a9e871e718581@tracker.bro-ids.org> #655: broctl 'scripts' command doesn't work in cluster mode -------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Problem | Status: closed Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: git/master Resolution: fixed | Keywords: -------------------------+------------------------ Changes (by jsiwek): * owner: => jsiwek * status: new => closed * resolution: => fixed Comment: In [90eb30481197ffe630a4fbe800d268be55d727c4/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="90eb30481197ffe630a4fbe800d268be55d727c4" Fix broctl 'scripts' command in cluster mode (fixes #655) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From vallentin at icir.org Mon Nov 28 12:40:16 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Mon, 28 Nov 2011 12:40:16 -0800 Subject: [Bro-Dev] string_cat(...) vs. cat(...) Message-ID: <1322512646-sup-2243@samurai.local> Is there a use case when using string_cat is a better fit than cat? The domain of the former is a subset of the latter, so I could only imagine that performance reasons may justify using it. Matthias From vallentin at icir.org Mon Nov 28 12:44:47 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Mon, 28 Nov 2011 12:44:47 -0800 Subject: [Bro-Dev] BiF prefix In-Reply-To: <20111128052258.25FCD2C4002@rock.ICSI.Berkeley.EDU> References: <20111128052258.25FCD2C4002@rock.ICSI.Berkeley.EDU> Message-ID: <1322513027-sup-369@samurai.local> > And maybe change it to have "file_mode" or "Unix_file_mode" in the name. Gregor, would you be okay with renaming the BiF to file_mode? Matthias From bro at tracker.bro-ids.org Mon Nov 28 13:25:08 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 28 Nov 2011 21:25:08 -0000 Subject: [Bro-Dev] #676: Broctl not relinking logs/current correctly In-Reply-To: <046.cd0ffbb5c01e8b46c16ffc30a036451e@tracker.bro-ids.org> References: <046.cd0ffbb5c01e8b46c16ffc30a036451e@tracker.bro-ids.org> Message-ID: <061.510151c32bfed81bb521d354c367c9c7@tracker.bro-ids.org> #676: Broctl not relinking logs/current correctly -------------------------+-------------------- Reporter: seth | Owner: jsiwek Type: Problem | Status: closed Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: Resolution: fixed | Keywords: -------------------------+-------------------- Changes (by jsiwek): * owner: => jsiwek * status: new => closed * resolution: => fixed Comment: In [acc0250179dd25accb93704cf01cfd375e778aa6/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="acc0250179dd25accb93704cf01cfd375e778aa6" Fix standalone->cluster upgrade failing to update logs/current symlink. Fixes #676. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Mon Nov 28 14:54:40 2011 From: gregor at icir.org (Gregor Maier) Date: Mon, 28 Nov 2011 14:54:40 -0800 Subject: [Bro-Dev] BiF prefix In-Reply-To: <1322455365-sup-5942@samurai.local> References: <1322455365-sup-5942@samurai.local> Message-ID: <4ED41130.80703@icir.org> On 11/27/11 20:44 , Matthias Vallentin wrote: > Why does the BiF > > NFS3::mode2string%(mode: count%): string > > have a prefix of NFS3? No other BiF has such a prefix. I added namespace support for BiF's with my work on NFS/RPC(*) and this one is the only new BiF I wrote. The idea is that in the long run we should migrate BiF's to appropriate namespaces. cu Gregor (*) That's when I also added support for making types, globals, enums, etc. easily accessible in C++, BiF, and script land -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From gregor at icir.org Mon Nov 28 14:58:11 2011 From: gregor at icir.org (Gregor Maier) Date: Mon, 28 Nov 2011 14:58:11 -0800 Subject: [Bro-Dev] BiF prefix In-Reply-To: <1322513027-sup-369@samurai.local> References: <20111128052258.25FCD2C4002@rock.ICSI.Berkeley.EDU> <1322513027-sup-369@samurai.local> Message-ID: <4ED41203.1060405@icir.org> On 11/28/11 12:44 , Matthias Vallentin wrote: >> And maybe change it to have "file_mode" or "Unix_file_mode" in the name. > > Gregor, would you be okay with renaming the BiF to file_mode? Should work. I actually can't recall off the top of my head whether the file mode encoding used by NFS is indeed normal Unix modes. If so, go ahead and change it. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Mon Nov 28 14:59:57 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 28 Nov 2011 22:59:57 -0000 Subject: [Bro-Dev] #698: HTTP vs MIME events Message-ID: <047.7f6f47529b886fee475b591da0309f6c@tracker.bro-ids.org> #698: HTTP vs MIME events ---------------------+----------------- Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: Keywords: cleanup | ---------------------+----------------- The way the ``mime_*`` and ``http_entity_*`` events are structure is a mess: - First, I think we should have only a single set of events for all MIME data. - the ``mime_*`` events don't come with ``is_orig``. - ``http_header`` vs. ``mime_one_header`` - ``http_entity_data`` delivers segments of size ``http_entity_data_delivery_size`` while ``mime_entity_data`` delivers complete entities (and ``mime_segment_data`` delivers segments?) - There are further inconsistencies I didn't record. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Nov 28 15:04:22 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 28 Nov 2011 23:04:22 -0000 Subject: [Bro-Dev] #699: Reorganizing layout of protocol analyzers Message-ID: <047.bdd8ceb4ce1b04ea31b0648ad0c67b84@tracker.bro-ids.org> #699: Reorganizing layout of protocol analyzers ------------------------+-------------------- Reporter: robin | Type: Task Status: new | Priority: Normal Milestone: Bro2.1 | Component: Bro Version: git/master | ------------------------+-------------------- We should restructure how protocol analyzers are organized inside the C++ core. Ideally, I'd like to have a single directory per analyzer with everything in there that relates to that analyzer, with no need to touch other places (currently, one needs to touch quite many for adding an analyzer). This includes the analyzers' ``*.cc``, ``*.h``, ``*.bif``, and ``*.pac`` files. For the ``bif``s, we should also move the corresponding parts from ``event.bif`` and ``init-bare.bif`` into that directory. This restructuring is also a good opportuntity to reorganize the Broxygen generation for analyzers: I think it would be nice to have one page per analyzer that has (1) a general description, (2) all tuning parameters, (3) all core events, and (4) links to all relevant ``base`` and ``policy`` scripts. I'm setting this to 2.1 for now. It's not high-priority for that release and we can bump it further out if necessary. But I think it's good to keep in mind, and perhaps work on parts of this as time permits. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Nov 28 15:06:42 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 28 Nov 2011 23:06:42 -0000 Subject: [Bro-Dev] #685: Change default constant for empty or unset field of LogWriterAscii In-Reply-To: <048.4a3ac67b40f5405235e87970b76f67ce@tracker.bro-ids.org> References: <048.4a3ac67b40f5405235e87970b76f67ce@tracker.bro-ids.org> Message-ID: <063.c5f6325e17e19319f2ec9e210ac58be2@tracker.bro-ids.org> #685: Change default constant for empty or unset field of LogWriterAscii ----------------------+-------------------- Reporter: amannb | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by amannb): I also think that having completely empty fields makes things hard to read. How does using [] for empty fields sound? -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Tue Nov 29 08:16:19 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 29 Nov 2011 08:16:19 -0800 Subject: [Bro-Dev] Questions about dns.log Message-ID: <20111129161619.GA62930@icir.org> Two questions about dns.log: - it logs all the answers for each request but only one TTL. Should "TTL" be a list of all the values in the corresponding order? - it doesn't log the type of the answers (A, MX, etc), which seems could be helpful? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From slagell at illinois.edu Tue Nov 29 08:23:28 2011 From: slagell at illinois.edu (Slagell, Adam J) Date: Tue, 29 Nov 2011 16:23:28 +0000 Subject: [Bro-Dev] Questions about dns.log In-Reply-To: <20111129161619.GA62930@icir.org> References: <20111129161619.GA62930@icir.org> Message-ID: <04D39102-EC6A-458E-9FA7-DE8154CE8279@illinois.edu> On Nov 29, 2011, at 10:16 AM, Robin Sommer wrote: > - it doesn't log the type of the answers (A, MX, etc), which seems > could be helpful? I thought that was odd myself going through and testing traces. From bro at tracker.bro-ids.org Tue Nov 29 09:32:19 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 29 Nov 2011 17:32:19 -0000 Subject: [Bro-Dev] #699: Reorganizing layout of protocol analyzers In-Reply-To: <047.bdd8ceb4ce1b04ea31b0648ad0c67b84@tracker.bro-ids.org> References: <047.bdd8ceb4ce1b04ea31b0648ad0c67b84@tracker.bro-ids.org> Message-ID: <062.79adea4f12928dc4a88e119957c2f3da@tracker.bro-ids.org> #699: Reorganizing layout of protocol analyzers ---------------------+------------------------ Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro2.1 Component: Bro | Version: git/master Resolution: | Keywords: ---------------------+------------------------ Comment (by gregor): {{{ #!rst I like the idea. However, the trickier part is IMHO the way analyzers have to be added in other places. In particular ``AnalyzerTags.h``, ``Analyzer.h``, and ``Analyzer.cc``. We should solve this. Eventually, this might enable us to have a plug-in interface for analyzers in which analyzers can be compiled as .so files and then loaded with dlopen(). Hopefully, this makes writing (and later integrating) analyzer for external developers easier. Regarding BiF: If an analyzer comes with BiF's then, the files generated by ``bifcl`` (there are 5 or so) need to be included in the appropriate places in the source code. This can probably be automated. E.g., consider the ``*.bif.func_def`` files. In the ``.cc/.h`` file we include a ``FOOBAR.func_def`` , which is automatically generated by the build system based and in turn has ``#includes`` for all generated ``*.bif.func_def``. This would also allow us to more easily split ``bro.bif`` in smaller chunks. Right now pretty much everything goes in there. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Nov 29 09:36:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 29 Nov 2011 17:36:50 -0000 Subject: [Bro-Dev] #700: PacketSorter Message-ID: <048.3b6beb7959a1599da5788f14c206fd37@tracker.bro-ids.org> #700: PacketSorter -------------------------+-------------------- Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.1 Component: Bro | Version: Keywords: BroV6, IPv6 | -------------------------+-------------------- (from an e-mail I sent a while ago) Might relevant for IPv6 so setting milestone to 2.1 Hi, I was wondering about Bro's packet sorter. From a quick glance it appears that it's only enabled if packet_sort_window is set to a non zero value. When enabled it will sort packets a) based on timestamps and b) for TCP packets based on SEQ/ACK numbers (I presume to ensure that ACKs are delivered after the data packet) Note, this is independent from Bro's ability to process multiple trace files (or multiple interfaces) in order. So I was wondering about the use cases for PacketSorter, especially (a) If the packet sorter is enabled Bro's behavior will slightly change: It won't pass ARP packets to the ARP analyzer, and it won't create a weird if it's not an IP packet. I was just wondering whether anybody has recently used the packet sorter. If not I'm wondering whether we should test this code path to see whether it works correctly esp wrt IPv6. Or, actually, whether the packet sorter is worth keeping or whether we should remove the code. And another question would be if the TCP sorting would better be handled by the TCP analyzer? Opinions? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Nov 29 09:38:22 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 29 Nov 2011 17:38:22 -0000 Subject: [Bro-Dev] #701: Autodoc final version of redef'ed records (for logging) Message-ID: <048.f55cbfb9e2a966664745393c7e203879@tracker.bro-ids.org> #701: Autodoc final version of redef'ed records (for logging) ---------------------+-------------------- Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.1 Component: Bro | Version: Keywords: | ---------------------+-------------------- (from an earlier e-mail) Hi, given a bunch of scripts, that extend record types (I mostly thinking about the Info records used for logging) if there a way to get a autodoc generated file that shows the final version of the record? The idea would be, that this "final version" is exactly what actually gets logged, so it would be a nice-to-have for users who want to understand all the columns in a log file. Now, they have to know all the scripts that are loaded that extend a particular record and look at their documentation to find out the meaning of columns / fields. This is fine for scripts in policy, but for the base scripts it would be nice to have a single place to go too. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Tue Nov 29 09:41:00 2011 From: seth at icir.org (Seth Hall) Date: Tue, 29 Nov 2011 12:41:00 -0500 Subject: [Bro-Dev] Questions about dns.log In-Reply-To: <20111129161619.GA62930@icir.org> References: <20111129161619.GA62930@icir.org> Message-ID: <43DE30FD-1138-49E0-8AAC-FC6DD086EC81@icir.org> On Nov 29, 2011, at 11:16 AM, Robin Sommer wrote: > Two questions about dns.log: I haven't ever really been satisfied with the content of that log. It's really hard to represent the DNS request/response pair though considering that you have to weigh data volume with typical use cases. It seems that all most people want (from a security forensics perspective) are the answers that came back from queries and who made the query. The technical details (even response type) matter surprisingly little in most cases. I would definitely like to talk about it more though. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Tue Nov 29 09:43:38 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 29 Nov 2011 09:43:38 -0800 Subject: [Bro-Dev] Sphinx: Linking to logs? Message-ID: <20111129174338.GE62930@icir.org> Do we have a way of linking to a log file? Say I want to write somethig like "this information goes into column X of conn.log". Or should I just link to the Conn::Info identifier? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Tue Nov 29 09:45:31 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 29 Nov 2011 09:45:31 -0800 Subject: [Bro-Dev] Questions about dns.log In-Reply-To: <43DE30FD-1138-49E0-8AAC-FC6DD086EC81@icir.org> References: <20111129161619.GA62930@icir.org> <43DE30FD-1138-49E0-8AAC-FC6DD086EC81@icir.org> Message-ID: <20111129174531.GF62930@icir.org> On Tue, Nov 29, 2011 at 12:41 -0500, you wrote: > the query. The technical details (even response type) matter > surprisingly little in most cases. Maybe, but right now at least the TTL is just misleading/confusing I think as it's not clear what it belongs to. > I would definitely like to talk about it more though. We can do that. :) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gregor at icir.org Tue Nov 29 09:49:56 2011 From: gregor at icir.org (Gregor Maier) Date: Tue, 29 Nov 2011 09:49:56 -0800 Subject: [Bro-Dev] Sphinx: Linking to logs? In-Reply-To: <20111129174338.GE62930@icir.org> References: <20111129174338.GE62930@icir.org> Message-ID: <4ED51B44.5070602@icir.org> On 11/29/11 9:43 , Robin Sommer wrote: > Do we have a way of linking to a log file? Say I want to write > somethig like "this information goes into column X of conn.log". Or > should I just link to the Conn::Info identifier? I would link to Conn::Info. After all, logging can be modified by the use using filters and/or different backends. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From jsiwek at illinois.edu Tue Nov 29 09:52:26 2011 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Tue, 29 Nov 2011 17:52:26 +0000 Subject: [Bro-Dev] Sphinx: Linking to logs? In-Reply-To: <20111129174338.GE62930@icir.org> References: <20111129174338.GE62930@icir.org> Message-ID: <64A26257-53DD-40E8-8883-DE202E82C06C@illinois.edu> > Do we have a way of linking to a log file? Say I want to write > somethig like "this information goes into column X of conn.log". Or > should I just link to the Conn::Info identifier? The only thing you can link to right now is the Conn::Info. +Jon From seth at icir.org Tue Nov 29 10:19:44 2011 From: seth at icir.org (Seth Hall) Date: Tue, 29 Nov 2011 13:19:44 -0500 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/script-reference: Almost done with event.bif. (0523a18) In-Reply-To: <201111291813.pATIDnWB015690@bro-ids.icir.org> References: <201111291813.pATIDnWB015690@bro-ids.icir.org> Message-ID: On Nov 29, 2011, at 1:13 PM, Robin Sommer wrote: > Repository : ssh://git at bro-ids.icir.org/bro > Almost done with event.bif. Awesome! > Added comments to almost all events. The only ones I'm leaving out are > Gnutella and BitTorrent, don't know enough about those protocols to > document the events. If anybody does, please chime in. I'm planning on modifying those events slightly for the 2.1 release anyway, I'll document them then. Are you ok just leaving them without documentation for now? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20111129/7162f3d2/attachment-0001.bin From robin at icir.org Tue Nov 29 10:23:37 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 29 Nov 2011 10:23:37 -0800 Subject: [Bro-Dev] Deprecating events Message-ID: <20111129182337.GG62930@icir.org> I'd like to deprecate the events below for 2.0, with the goal to remove them for 2.1. Any objections? Reason for deprecation is that these (1) are hardly used anywhere anymore; and/or (2) use a model that doesn't fit well with how other scripts are structured (like: do a lot of stuff inside the event engine); and/or (3) have better ways to do it now (like the backdoor events: signatures and DPD). For the file-related events, I'm thinking that with the new logging framework, we should remove all the "extra" functionality from script-level files that we have added over time (rotation, postprocessing, etc). The log framework provides all of that and is the preferred way of doing it. Having two separate mechanisms for the same functionality doesn't seem good. (To be clear, I want to keep files themselves as a script-level data type, but without all the bells and whistles) There are more events that fit (1)-(3), in particular the pattern-matching login_* events. Undecided whether those should go too, but I have documented them for now. Robin ------------------------- ## Deprecated. Will be removed. event stp_create_endp%(c: connection, e: int, is_orig: bool%); ## Deprecated. Will be removed. event stp_resume_endp%(e: int%); ## Deprecated. Will be removed. event stp_correlate_pair%(e1: int, e2: int%); ## Deprecated. Will be removed. event stp_remove_pair%(e1: int, e2: int%); ## Deprecated. Will be removed. event stp_remove_endp%(e: int%); ## Deprecated. Will be removed. event interconn_stats%(c: connection, os: interconn_endp_stats, rs: interconn_endp_stats%); ## Deprecated. Will be removed. event interconn_remove_conn%(c: connection%); ## Deprecated. Will be removed. event backdoor_stats%(c: connection, os: backdoor_endp_stats, rs: backdoor_endp_stats%); ## Deprecated. Will be removed. event backdoor_remove_conn%(c: connection%); ## Deprecated. Will be removed. event ssh_signature_found%(c: connection, is_orig: bool%); ## Deprecated. Will be removed. event telnet_signature_found%(c: connection, is_orig: bool, len: count%); ## Deprecated. Will be removed. event rlogin_signature_found%(c: connection, is_orig: bool, num_null: count, len: count%); ## Deprecated. Will be removed. event root_backdoor_signature_found%(c: connection%); ## Deprecated. Will be removed. event ftp_signature_found%(c: connection%); ## Deprecated. Will be removed. event napster_signature_found%(c: connection%); ## Deprecated. Will be removed. event gnutella_signature_found%(c: connection%); ## Deprecated. Will be removed. event kazaa_signature_found%(c: connection%); ## Deprecated. Will be removed. event http_signature_found%(c: connection%); ## Deprecated. Will be removed. event http_proxy_signature_found%(c: connection%); ## Deprecated. Will be removed. event smtp_signature_found%(c: connection%); ## Deprecated. Will be removed. event irc_signature_found%(c: connection%); ## Deprecated. Will be removed. event gaobot_signature_found%(c: connection%); ## Deprecated. Will be removed. ## ## .. todo:: Unclear what this event is for; it's never raised. We should just ## remove it. event dns_full_request%(%) &group="dns"; ## Deprecated. Will be removed. event anonymization_mapping%(orig: addr, mapped: addr%); ## Deprecated. Will be removed. event rotate_interval%(f: file%); ## Deprecated. Will be removed. event rotate_size%(f: file%); ## Deprecated. Will be removed. event print_hook%(f:file, s: string%); -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Tue Nov 29 10:28:00 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 29 Nov 2011 10:28:00 -0800 Subject: [Bro-Dev] Inactive events (Re: [Bro-Commits] [git/bro] topic/script-reference: Almost done with event.bif. (0523a18)) In-Reply-To: <201111291813.pATIDnWB015690@bro-ids.icir.org> References: <201111291813.pATIDnWB015690@bro-ids.icir.org> Message-ID: <20111129182800.GH62930@icir.org> On Tue, Nov 29, 2011 at 10:13 -0800, I wrote: > Almost done with event.bif. One question: a number of protocol events are not triggered right now because 2.0 doesn't have a correspondign script yet (and thus doesn't associate any ports with those analyzers). I'm not sure how to document that: I could add a ".. todo:: " to each event describing what's going on, but that looks suboptimal. Other ideas? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Tue Nov 29 10:28:39 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 29 Nov 2011 10:28:39 -0800 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/script-reference: Almost done with event.bif. (0523a18) In-Reply-To: References: <201111291813.pATIDnWB015690@bro-ids.icir.org> Message-ID: <20111129182839.GI62930@icir.org> On Tue, Nov 29, 2011 at 13:19 -0500, you wrote: > I'm planning on modifying those events slightly for the 2.1 release anyway, I'll document them then. Are you ok just leaving them without documentation for now? I am. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Tue Nov 29 10:35:10 2011 From: seth at icir.org (Seth Hall) Date: Tue, 29 Nov 2011 13:35:10 -0500 Subject: [Bro-Dev] Inactive events (Re: [Bro-Commits] [git/bro] topic/script-reference: Almost done with event.bif. (0523a18)) In-Reply-To: <20111129182800.GH62930@icir.org> References: <201111291813.pATIDnWB015690@bro-ids.icir.org> <20111129182800.GH62930@icir.org> Message-ID: <2140A0DB-4370-41F3-83E8-4F4283E451B1@icir.org> On Nov 29, 2011, at 1:28 PM, Robin Sommer wrote: > I'm not sure how to > document that: I could add a ".. todo:: " to each > event describing what's going on, but that looks suboptimal. I don't have a better idea, but since many of those protocol enabler scripts will be done for the 2.1 release (i hope!) I think that a suboptimal solution for this is actually ok. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20111129/54a60dde/attachment.bin From robin at icir.org Tue Nov 29 16:25:55 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 29 Nov 2011 16:25:55 -0800 Subject: [Bro-Dev] Empty log fields Message-ID: <20111130002555.GT62930@icir.org> We said we'd use "na" for unset record fields instead of "-", but now I'm pondering over what version of that *exactly*: (1) NA (2) N/A (3) na (4) n/a I'm voting for (4). Other opinions? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bernhard at ICSI.Berkeley.EDU Tue Nov 29 16:53:21 2011 From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann) Date: Tue, 29 Nov 2011 16:53:21 -0800 Subject: [Bro-Dev] Call for opinions on logging framework syntax problem Message-ID: <40827F30-AF22-4993-885D-0A9C6B25C499@icsi.berkeley.edu> Hello, I am currently working on the input framework for bro -- which allows reading previously written log files back into bro -- and have encountered a little problem when reading port fields. There are several different methods to solve this problem and I wanted to get a little bit of feedback before implementing any of these solutions. First to describe the problem... When the logging framework is used to log port information, the information does not include the protocol -- this is usually stored in a second column. Hence, a logfile storing port information will usually look something like this #fields host_a host_p proto 12.12.12.12 80 tcp The input framework uses record types to define, what fields should be read from a previously written logfile. To read the fields, one could e.g. define a record like this: type Values: record { host_a: addr; host_p: port; } The problem with this approach is, that now host_p does not contain the protocol of the port, because it is stored in a different, unrelated column. Hence, the input framework needs a (preferably syntactically nice, easy to understand) way to identify the column that is used to store the port information. The easiest solution would be just to assume a fixed column name (e.g. host_p_port if the port is stored in host_p), but this is probably not a very good idea for a number of reasons. The nicest way we could think of at the moment is to use annotations for this; for our example one could e.g. use type Values: record { host_a: addr; host_p: port &protocol_column=proto; } This has the disadvantage of introducing a new, very specialized annotation that is only used for this one case. Does anyone else have any ideas / suggestions? Bernhard From vallentin at icir.org Tue Nov 29 16:56:56 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Tue, 29 Nov 2011 16:56:56 -0800 Subject: [Bro-Dev] Empty log fields In-Reply-To: <20111130002555.GT62930@icir.org> References: <20111130002555.GT62930@icir.org> Message-ID: > ? ?(1) NA > ? ?(2) N/A > ? ?(3) na > ? ?(4) n/a I'm voting for (1). (Sorry Robin :-) Matthias From slagell at illinois.edu Tue Nov 29 16:59:25 2011 From: slagell at illinois.edu (Slagell, Adam J) Date: Wed, 30 Nov 2011 00:59:25 +0000 Subject: [Bro-Dev] Empty log fields In-Reply-To: References: <20111130002555.GT62930@icir.org>, Message-ID: <2E54C1AF-717D-426C-A410-17F29366154F@illinois.edu> I vote 2. :-) isn't democracy grand? Sent from my mobile On Nov 29, 2011, at 6:57 PM, "Matthias Vallentin" wrote: >> (1) NA >> (2) N/A >> (3) na >> (4) n/a > > I'm voting for (1). > > (Sorry Robin :-) > > Matthias > > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > From seth at icir.org Tue Nov 29 17:00:10 2011 From: seth at icir.org (Seth Hall) Date: Tue, 29 Nov 2011 20:00:10 -0500 Subject: [Bro-Dev] Empty log fields In-Reply-To: <20111130002555.GT62930@icir.org> References: <20111130002555.GT62930@icir.org> Message-ID: <9E6A5837-1531-4ED2-A2BD-864890B39A32@icir.org> On Nov 29, 2011, at 7:25 PM, Robin Sommer wrote: > (4) n/a > > I'm voting for (4). Other opinions? I think I like #4 best too. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20111129/5000abeb/attachment.bin From seth at icir.org Tue Nov 29 17:07:35 2011 From: seth at icir.org (Seth Hall) Date: Tue, 29 Nov 2011 20:07:35 -0500 Subject: [Bro-Dev] Empty log fields In-Reply-To: <2E54C1AF-717D-426C-A410-17F29366154F@illinois.edu> References: <20111130002555.GT62930@icir.org>, <2E54C1AF-717D-426C-A410-17F29366154F@illinois.edu> Message-ID: On Nov 29, 2011, at 7:59 PM, Slagell, Adam J wrote: > I vote 2. :-) isn't democracy grand? Haha, now that we're getting a nice spread of opinions on this, I'll toss a couple more in the pot? 5. nil 6. null Since this value really represents a value not being set, nil/null would make sense. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20111129/70520fad/attachment.bin From slagell at illinois.edu Tue Nov 29 17:11:08 2011 From: slagell at illinois.edu (Slagell, Adam J) Date: Wed, 30 Nov 2011 01:11:08 +0000 Subject: [Bro-Dev] Empty log fields In-Reply-To: References: <20111130002555.GT62930@icir.org>, <2E54C1AF-717D-426C-A410-17F29366154F@illinois.edu>, Message-ID: $\{\phi\}$ for the empty set Sent from my mobile On Nov 29, 2011, at 7:07 PM, "Seth Hall" wrote: > > On Nov 29, 2011, at 7:59 PM, Slagell, Adam J wrote: > >> I vote 2. :-) isn't democracy grand? > > > Haha, now that we're getting a nice spread of opinions on this, I'll toss a couple more in the pot? > > 5. nil > 6. null > > Since this value really represents a value not being set, nil/null would make sense. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > From vallentin at icir.org Tue Nov 29 17:54:21 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Tue, 29 Nov 2011 17:54:21 -0800 Subject: [Bro-Dev] Empty log fields In-Reply-To: References: <20111130002555.GT62930@icir.org> <2E54C1AF-717D-426C-A410-17F29366154F@illinois.edu> Message-ID: On Tue, Nov 29, 2011 at 5:11 PM, Slagell, Adam J wrote: > $\{\phi\}$ for the empty set The correct LaTeX symbol for the empty set is actually $\emptyset$. Let's have UTF-8 logs so that we can use "?" (U+2205) for the empty set ;-). SCNR, Matthias From robin at icir.org Tue Nov 29 18:12:53 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 29 Nov 2011 18:12:53 -0800 Subject: [Bro-Dev] Empty log fields In-Reply-To: References: <20111130002555.GT62930@icir.org> <2E54C1AF-717D-426C-A410-17F29366154F@illinois.edu> Message-ID: <20111130021253.GX62930@icir.org> On Tue, Nov 29, 2011 at 17:54 -0800, you wrote: > The correct LaTeX symbol for the empty set is actually $\emptyset$. Let's have > UTF-8 logs so that we can use "?" (U+2205) for the empty set ;-). We could also base64 encode a transparent 1x1 PNG image and insert that there ... Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gregor at icir.org Tue Nov 29 19:04:00 2011 From: gregor at icir.org (Gregor Maier) Date: Tue, 29 Nov 2011 19:04:00 -0800 Subject: [Bro-Dev] Call for opinions on logging framework syntax problem In-Reply-To: <40827F30-AF22-4993-885D-0A9C6B25C499@icsi.berkeley.edu> References: <40827F30-AF22-4993-885D-0A9C6B25C499@icsi.berkeley.edu> Message-ID: <4ED59D20.9050805@icir.org> I'm wondering whether we should maybe start including the protocol in the same column in the log files. I.e., the column would then be "80/tcp".... Or we could handle ports similar to embedded records in the log file. I.e., if we log a port variable named "orig_p" we would get two columns: orig_p.port orig_p.proto I actually like this variant! cu Gregor On 11/29/11 16:53 , Bernhard Amann wrote: > Hello, > > I am currently working on the input framework for bro -- which allows reading previously written log files back into bro -- and have encountered a little problem when reading port fields. There are several different methods to solve this problem and I wanted to get a little bit of feedback before implementing any of these solutions. > > First to describe the problem... > When the logging framework is used to log port information, the information does not include the protocol -- this is usually stored in a second column. > Hence, a logfile storing port information will usually look something like this > > #fields host_a host_p proto > 12.12.12.12 80 tcp > > The input framework uses record types to define, what fields should be read from a previously written logfile. > To read the fields, one could e.g. define a record like this: > > type Values: record { > host_a: addr; > host_p: port; > } > > The problem with this approach is, that now host_p does not contain the protocol of the port, because it is stored in a different, unrelated column. > Hence, the input framework needs a (preferably syntactically nice, easy to understand) way to identify the column that is used to store the port information. > > The easiest solution would be just to assume a fixed column name (e.g. host_p_port if the port is stored in host_p), but this is probably not a very good idea for a number of reasons. > > The nicest way we could think of at the moment is to use annotations for this; for our example one could e.g. use > > type Values: record { > host_a: addr; > host_p: port&protocol_column=proto; > } > > This has the disadvantage of introducing a new, very specialized annotation that is only used for this one case. > > Does anyone else have any ideas / suggestions? > > Bernhard > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bernhard at ICSI.Berkeley.EDU Tue Nov 29 20:06:26 2011 From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann) Date: Tue, 29 Nov 2011 20:06:26 -0800 Subject: [Bro-Dev] Call for opinions on logging framework syntax problem In-Reply-To: <4ED59D20.9050805@icir.org> References: <40827F30-AF22-4993-885D-0A9C6B25C499@icsi.berkeley.edu> <4ED59D20.9050805@icir.org> Message-ID: Adding the protocol to the port column in the logging framework would be a (for me) very easy solution - and probably also the cleanest one. However, apparently that is not desired, because it makes the port field more difficult to parse. It also would fix another problem. At the moment is possible to have sets or vectors of ports in a single line of the log file. That could e.g. be used to note several ports a host is active on, e.g. #fields host services 12.12.12.12 53,80,8080 When adding the protocol directly to the port information, the log line would e.g. look like 12.12.12.12 53/udp,80/tcp,8080/tcp When using a second column one could probably do something like 12.12.12.12 53,80,8080 udp,tcp,tcp but that is not really easy to read. Bernhard On Nov 29, 2011, at 7:04 PM, Gregor Maier wrote: > I'm wondering whether we should maybe start including the protocol in the same column in the log files. I.e., the column would then be "80/tcp".... > > Or we could handle ports similar to embedded records in the log file. I.e., if we log a port variable named "orig_p" we would get two columns: > orig_p.port orig_p.proto > I actually like this variant! > > > cu > Gregor > > On 11/29/11 16:53 , Bernhard Amann wrote: >> Hello, >> >> I am currently working on the input framework for bro -- which allows reading previously written log files back into bro -- and have encountered a little problem when reading port fields. There are several different methods to solve this problem and I wanted to get a little bit of feedback before implementing any of these solutions. >> >> First to describe the problem... >> When the logging framework is used to log port information, the information does not include the protocol -- this is usually stored in a second column. >> Hence, a logfile storing port information will usually look something like this >> >> #fields host_a host_p proto >> 12.12.12.12 80 tcp >> >> The input framework uses record types to define, what fields should be read from a previously written logfile. >> To read the fields, one could e.g. define a record like this: >> >> type Values: record { >> host_a: addr; >> host_p: port; >> } >> >> The problem with this approach is, that now host_p does not contain the protocol of the port, because it is stored in a different, unrelated column. >> Hence, the input framework needs a (preferably syntactically nice, easy to understand) way to identify the column that is used to store the port information. >> >> The easiest solution would be just to assume a fixed column name (e.g. host_p_port if the port is stored in host_p), but this is probably not a very good idea for a number of reasons. >> >> The nicest way we could think of at the moment is to use annotations for this; for our example one could e.g. use >> >> type Values: record { >> host_a: addr; >> host_p: port&protocol_column=proto; >> } >> >> This has the disadvantage of introducing a new, very specialized annotation that is only used for this one case. >> >> Does anyone else have any ideas / suggestions? >> >> Bernhard >> _______________________________________________ >> bro-dev mailing list >> bro-dev at bro-ids.org >> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > > > -- > Gregor Maier > > Int. Computer Science Institute (ICSI) > 1947 Center St., Ste. 600 > Berkeley, CA 94704, USA > http://www.icir.org/gregor/ From seth at icir.org Tue Nov 29 20:19:14 2011 From: seth at icir.org (Seth Hall) Date: Tue, 29 Nov 2011 23:19:14 -0500 Subject: [Bro-Dev] Call for opinions on logging framework syntax problem In-Reply-To: <4ED59D20.9050805@icir.org> References: <40827F30-AF22-4993-885D-0A9C6B25C499@icsi.berkeley.edu> <4ED59D20.9050805@icir.org> Message-ID: On Nov 29, 2011, at 10:04 PM, Gregor Maier wrote: > I'm wondering whether we should maybe start including the protocol in > the same column in the log files. I.e., the column would then be > "80/tcp"?. I don't like this since no databases have an analogous data type and integers are nice and searchable. > Or we could handle ports similar to embedded records in the log file. > I.e., if we log a port variable named "orig_p" we would get two columns: > orig_p.port orig_p.proto > I actually like this variant! I like this too, but I get the sense that the protocol should actually be an attribute of the conn_id type and not a part of each port value. If we started using counts for port values (and get rid of the port type?) and add a $proto field to conn_id does that break any existing assumptions within the language? There are a number of cases where the port type has caused me grief for various reasons but I'm not sure if there is some deeper functionality I'm missing that we would lack with this change. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20111129/abb5be77/attachment.bin From seth at icir.org Tue Nov 29 20:24:31 2011 From: seth at icir.org (Seth Hall) Date: Tue, 29 Nov 2011 23:24:31 -0500 Subject: [Bro-Dev] Call for opinions on logging framework syntax problem In-Reply-To: References: <40827F30-AF22-4993-885D-0A9C6B25C499@icsi.berkeley.edu> <4ED59D20.9050805@icir.org> Message-ID: <6E810C07-A416-43C8-AF60-4492EE66E647@icir.org> On Nov 29, 2011, at 11:06 PM, Bernhard Amann wrote: > When adding the protocol directly to the port information, the log line would e.g. look like > > 12.12.12.12 53/udp,80/tcp,8080/tcp This is definitely one place where the email I just sent breaks down. It's the port value used outside of the context of a conn_id value. Do you have a concrete example of when you'd want to do something like this? I suspect that if you wanted to do that it would actually be better to organize your data in a different way. Like this: #fields host port proto 12.12.12.12 53 udp 12.12.12.12 80 tcp 12.12.12.12 8080 tcp .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20111129/978e7bd4/attachment.bin From bernhard at ICSI.Berkeley.EDU Tue Nov 29 21:19:50 2011 From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann) Date: Tue, 29 Nov 2011 21:19:50 -0800 Subject: [Bro-Dev] Call for opinions on logging framework syntax problem In-Reply-To: <6E810C07-A416-43C8-AF60-4492EE66E647@icir.org> References: <40827F30-AF22-4993-885D-0A9C6B25C499@icsi.berkeley.edu> <4ED59D20.9050805@icir.org> <6E810C07-A416-43C8-AF60-4492EE66E647@icir.org> Message-ID: <39BED8D0-75D9-44A9-B2D4-41ACF774B61E@ICSI.Berkeley.EDU> No, I have no real concrete example? I just tried to think of things people might perhaps want to do. And the use-case of having a set of ports for one IP did not seem too far fetched. Bernhard On Nov 29, 2011, at 8:24 PM, Seth Hall wrote: > > On Nov 29, 2011, at 11:06 PM, Bernhard Amann wrote: > >> When adding the protocol directly to the port information, the log line would e.g. look like >> >> 12.12.12.12 53/udp,80/tcp,8080/tcp > > > This is definitely one place where the email I just sent breaks down. It's the port value used outside of the context of a conn_id value. Do you have a concrete example of when you'd want to do something like this? I suspect that if you wanted to do that it would actually be better to organize your data in a different way. Like this: > > #fields host port proto > 12.12.12.12 53 udp > 12.12.12.12 80 tcp > 12.12.12.12 8080 tcp > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > From bro at tracker.bro-ids.org Tue Nov 29 22:13:46 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 30 Nov 2011 06:13:46 -0000 Subject: [Bro-Dev] #656: Add stats.bro back In-Reply-To: <047.2d9bdf13c6cee04531b4645e09da8cb7@tracker.bro-ids.org> References: <047.2d9bdf13c6cee04531b4645e09da8cb7@tracker.bro-ids.org> Message-ID: <062.6831f5834bd8f8287ac1d7ae1374567d@tracker.bro-ids.org> #656: Add stats.bro back ---------------------+------------------------ Reporter: robin | Owner: seth Type: Task | Status: assigned Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: git/master Resolution: | Keywords: ---------------------+------------------------ Changes (by seth): * owner: => seth * status: new => assigned -- Ticket URL: Bro Tracker Bro Issue Tracker From gc355804 at ohio.edu Tue Nov 29 22:35:45 2011 From: gc355804 at ohio.edu (G. Clark) Date: Wed, 30 Nov 2011 01:35:45 -0500 Subject: [Bro-Dev] Empty log fields In-Reply-To: <20111130021253.GX62930@icir.org> References: <20111130002555.GT62930@icir.org> <2E54C1AF-717D-426C-A410-17F29366154F@illinois.edu> <20111130021253.GX62930@icir.org> Message-ID: <4ED5CEC1.4040707@ohio.edu> I propose a haiku: the data given packet form never here More seriously: n/a would break a scheme that used '/' as a separator character, and would be a little tougher to deal wit On 11/29/11 9:12 PM, Robin Sommer wrote: > > On Tue, Nov 29, 2011 at 17:54 -0800, you wrote: > >> The correct LaTeX symbol for the empty set is actually $\emptyset$. Let's have >> UTF-8 logs so that we can use "?" (U+2205) for the empty set ;-). > > We could also base64 encode a transparent 1x1 PNG image and insert > that there ... > > Robin > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6010 bytes Desc: S/MIME Cryptographic Signature Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20111130/eeb2da07/attachment-0001.bin From gc355804 at ohio.edu Tue Nov 29 22:40:53 2011 From: gc355804 at ohio.edu (G. Clark) Date: Wed, 30 Nov 2011 01:40:53 -0500 Subject: [Bro-Dev] Empty log fields In-Reply-To: <4ED5CEC1.4040707@ohio.edu> References: <20111130002555.GT62930@icir.org> <2E54C1AF-717D-426C-A410-17F29366154F@illinois.edu> <20111130021253.GX62930@icir.org> <4ED5CEC1.4040707@ohio.edu> Message-ID: <4ED5CFF5.7070405@ohio.edu> *, and would be a little tougher to deal with than null / nil / other character strings, I believe. --Gilbert On 11/30/11 1:35 AM, G. Clark wrote: > I propose a haiku: > > the data > given packet form > never here > > More seriously: n/a would break a scheme that used '/' as a separator > character, and would be a little tougher to deal wit > > On 11/29/11 9:12 PM, Robin Sommer wrote: >> >> On Tue, Nov 29, 2011 at 17:54 -0800, you wrote: >> >>> The correct LaTeX symbol for the empty set is actually $\emptyset$. Let's have >>> UTF-8 logs so that we can use "?" (U+2205) for the empty set ;-). >> >> We could also base64 encode a transparent 1x1 PNG image and insert >> that there ... >> >> Robin >> > > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6010 bytes Desc: S/MIME Cryptographic Signature Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20111130/33229821/attachment.bin From bro at tracker.bro-ids.org Wed Nov 30 07:30:12 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 30 Nov 2011 15:30:12 -0000 Subject: [Bro-Dev] #702: topic/seth/dns-updates - Fixed some DNS issues Message-ID: <046.2f4b2ad86a7b89075747e60a28802b6f@tracker.bro-ids.org> #702: topic/seth/dns-updates - Fixed some DNS issues ---------------------------+-------------------- Reporter: seth | Owner: robin Type: Merge Request | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Keywords: | ---------------------------+-------------------- This just needs merged. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Wed Nov 30 08:31:03 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 30 Nov 2011 08:31:03 -0800 Subject: [Bro-Dev] Call for opinions on logging framework syntax problem In-Reply-To: References: <40827F30-AF22-4993-885D-0A9C6B25C499@icsi.berkeley.edu> <4ED59D20.9050805@icir.org> Message-ID: <20111130163103.GD96269@icir.org> On Tue, Nov 29, 2011 at 23:19 -0500, you wrote: > If we started using counts for port values (and get rid of the port > type?) and add a $proto field to conn_id does that break any existing > assumptions within the language? Vern might know whether there's anything that would severely break but my guess is that in principle we could do that. However, I like the port type, it provides additional context for type checking and understanding a script. Also, changing that would be quite a large internal change: the type is used at many places and at some, we'd be now be left without port information where we need it. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Wed Nov 30 08:32:43 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 30 Nov 2011 08:32:43 -0800 Subject: [Bro-Dev] Call for opinions on logging framework syntax problem In-Reply-To: <4ED59D20.9050805@icir.org> References: <40827F30-AF22-4993-885D-0A9C6B25C499@icsi.berkeley.edu> <4ED59D20.9050805@icir.org> Message-ID: <20111130163243.GE96269@icir.org> On Tue, Nov 29, 2011 at 19:04 -0800, you wrote: > orig_p.port orig_p.proto Hmm ... I like that too but it breaks when ports are in sets/vectors, and it gets ugly with ports in records. We could do it just for top-level fields but that would be quite inconsistent. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Wed Nov 30 08:36:02 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 30 Nov 2011 16:36:02 -0000 Subject: [Bro-Dev] #633: Globals cleanup In-Reply-To: <047.cf4580f5113530897d7e17a5b9de40dc@tracker.bro-ids.org> References: <047.cf4580f5113530897d7e17a5b9de40dc@tracker.bro-ids.org> Message-ID: <062.705079dcdb17642f59a63b3499dc0602@tracker.bro-ids.org> #633: Globals cleanup ----------------------+------------------------ Reporter: robin | Owner: seth Type: Problem | Status: closed Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------+------------------------ Changes (by seth): * owner: => seth * status: new => closed * resolution: => fixed Comment: In [70004cb04d865fcc48aea11c05c01fe66a2b4f0d/bro]: {{{ #!CommitTicketReference repository="bro" revision="70004cb04d865fcc48aea11c05c01fe66a2b4f0d" Small updates to address the "globals" ticket. Fixes #633 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Wed Nov 30 08:41:17 2011 From: seth at icir.org (Seth Hall) Date: Wed, 30 Nov 2011 11:41:17 -0500 Subject: [Bro-Dev] Call for opinions on logging framework syntax problem In-Reply-To: <20111130163103.GD96269@icir.org> References: <40827F30-AF22-4993-885D-0A9C6B25C499@icsi.berkeley.edu> <4ED59D20.9050805@icir.org> <20111130163103.GD96269@icir.org> Message-ID: On Nov 30, 2011, at 11:31 AM, Robin Sommer wrote: > Vern might know whether there's anything that would severely break but > my guess is that in principle we could do that. However, I like the > port type, it provides additional context for type checking and > understanding a script I like it too, until I try to use it too much. :) It provides a very nice convenience within the language but creates a large complication if you try to use the values outside of the language. If you really think about it, it seems a little silly that we have to have a BiF for accessing the protocol of a port. If it was a record or a field in a conn_id we would be able to reuse existing knowledge to access the protocol. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20111130/a67f9ff4/attachment.bin From vern at icir.org Wed Nov 30 09:03:27 2011 From: vern at icir.org (Vern Paxson) Date: Wed, 30 Nov 2011 09:03:27 -0800 Subject: [Bro-Dev] Deprecating events In-Reply-To: <20111129182337.GG62930@icir.org> (Tue, 29 Nov 2011 10:23:37 PST). Message-ID: <20111130170327.F0DE22C409E@rock.ICSI.Berkeley.EDU> > ## Deprecated. Will be removed. > event stp_create_endp%(c: connection, e: int, is_orig: bool%); > ... Is the intent to remove the stepping stone detection functionality? That would be a pity, as now-and-then it provides very valuable forensic information. > ## Deprecated. Will be removed. > event interconn_stats%(c: connection, os: interconn_endp_stats, rs: interconn_endp_stats%); > ... > ## Deprecated. Will be removed. > event ssh_signature_found%(c: connection, is_orig: bool%); I agree with removing this stuff, as interconn never worked that well, and the signature stuff is all better done these days with DPD, or at least with um the signature engine. > There are more events that fit (1)-(3), in particular the > pattern-matching login_* events. Undecided whether those should go > too, but I have documented them for now. I'd be reluctant to lose these, as they could potentially become relevant if one is able to feed unencrypted SSH streams to Bro (depending on how the SSH server is set up). Vern From robin at icir.org Wed Nov 30 09:16:48 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 30 Nov 2011 09:16:48 -0800 Subject: [Bro-Dev] Deprecating events In-Reply-To: <20111130170327.F0DE22C409E@rock.ICSI.Berkeley.EDU> References: <20111129182337.GG62930@icir.org> <20111130170327.F0DE22C409E@rock.ICSI.Berkeley.EDU> Message-ID: <20111130171648.GI96269@icir.org> On Wed, Nov 30, 2011 at 09:03 -0800, you wrote: > Is the intent to remove the stepping stone detection functionality? Yes, that's what I was thinking. > That would be a pity, as now-and-then it provides very valuable forensic > information. I didn't realize this is still being used. I'm fine keeping the events then, but could you provide a few sentences describing their semantics for the script reference? I don't really know. > I'd be reluctant to lose these, as they could potentially become relevant > if one is able to feed unencrypted SSH streams to Bro That's right but isn't the scripting land the better place to implement this functionality eventually? What I don't like is all the hard-coded regexp variables that one passes into the core; that's quite different from any other analyzer. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From scampbell at lbl.gov Wed Nov 30 09:32:05 2011 From: scampbell at lbl.gov (Scott Campbell) Date: Wed, 30 Nov 2011 11:32:05 -0600 Subject: [Bro-Dev] Deprecating events In-Reply-To: <20111130170327.F0DE22C409E@rock.ICSI.Berkeley.EDU> References: <20111130170327.F0DE22C409E@rock.ICSI.Berkeley.EDU> Message-ID: <4ED66895.6030601@lbl.gov> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/30/11 11:03 AM, Vern Paxson wrote: >> ## Deprecated. Will be removed. event stp_create_endp%(c: >> connection, e: int, is_orig: bool%); ... > > Is the intent to remove the stepping stone detection > functionality? That would be a pity, as now-and-then it provides > very valuable forensic information. > >> ## Deprecated. Will be removed. event interconn_stats%(c: >> connection, os: interconn_endp_stats, rs: >> interconn_endp_stats%); ... ## Deprecated. Will be removed. event >> ssh_signature_found%(c: connection, is_orig: bool%); > > I agree with removing this stuff, as interconn never worked that > well, and the signature stuff is all better done these days with > DPD, or at least with um the signature engine. > >> There are more events that fit (1)-(3), in particular the >> pattern-matching login_* events. Undecided whether those should >> go too, but I have documented them for now. > > I'd be reluctant to lose these, as they could potentially become > relevant if one is able to feed unencrypted SSH streams to Bro > (depending on how the SSH server is set up). > > Vern _______________________________________________ bro-dev > mailing list bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev Re SSH streams, The iSSHD framework builds off the pattern matching login-* events but in that case we just take advantage of the policy infrastructure rather than the event generation. cheers, scott -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFO1miVK2Plq8B7ZBwRAlofAJ9ovBEYaGZvRtiVirq8kTGb/5jfggCeJhES Azkgrn6zALsn5Y5de24PdnU= =gm8W -----END PGP SIGNATURE----- From bro at tracker.bro-ids.org Wed Nov 30 11:40:30 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 30 Nov 2011 19:40:30 -0000 Subject: [Bro-Dev] #661: Crash in host name lookup In-Reply-To: <047.b51474059b3622f2b1778a88ef2d7190@tracker.bro-ids.org> References: <047.b51474059b3622f2b1778a88ef2d7190@tracker.bro-ids.org> Message-ID: <062.87fa68371c30fa5c59adb7abdaf5d226@tracker.bro-ids.org> #661: Crash in host name lookup ----------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by jsiwek): In [4444c56a9494fc3e38c1d39ae7d0d0fc812b5b8f/bro]: {{{ #!CommitTicketReference repository="bro" revision="4444c56a9494fc3e38c1d39ae7d0d0fc812b5b8f" Fix double-free of DNS_Mgr_Request object (addresses #661) In DNS::Resolve, they could be deleted once from where they were stored in the nb_dns_info cookie and once again from where they were stored in the DNS_Mgr::requests list. Before commit bd9c9372368f547d0930932de0995f50762b501c, they were only deleted from the requests list, so this commit reverts to that behavior without any leaks being reported by the core/leaks tests. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 30 11:42:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 30 Nov 2011 19:42:36 -0000 Subject: [Bro-Dev] #661: Crash in host name lookup In-Reply-To: <047.b51474059b3622f2b1778a88ef2d7190@tracker.bro-ids.org> References: <047.b51474059b3622f2b1778a88ef2d7190@tracker.bro-ids.org> Message-ID: <062.e766fd4df9e5a6b6b2bddaf590598999@tracker.bro-ids.org> #661: Crash in host name lookup ----------------------------+-------------------- Reporter: robin | Owner: Type: Merge Request | Status: new Priority: High | Milestone: Bro2.0 Component: Bro | Version: Resolution: | Keywords: ----------------------------+-------------------- Changes (by jsiwek): * type: Problem => Merge Request Comment: Ready to merge in `topic/jsiwek/fix-dns-double-free` -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Wed Nov 30 11:46:30 2011 From: gregor at icir.org (Gregor Maier) Date: Wed, 30 Nov 2011 11:46:30 -0800 Subject: [Bro-Dev] Call for opinions on logging framework syntax problem In-Reply-To: <20111130163243.GE96269@icir.org> References: <40827F30-AF22-4993-885D-0A9C6B25C499@icsi.berkeley.edu> <4ED59D20.9050805@icir.org> <20111130163243.GE96269@icir.org> Message-ID: <4ED68816.6000806@icir.org> On 11/30/11 8:32 , Robin Sommer wrote: > > On Tue, Nov 29, 2011 at 19:04 -0800, you wrote: > >> orig_p.port orig_p.proto > > Hmm ... I like that too but it breaks when ports are in sets/vectors, > and it gets ugly with ports in records. We could do it just for > top-level fields but that would be quite inconsistent. Yeah. I can see sets, vectors being a problem, however, wouldn't it break in the same way as sets/vectors of record types? For ports in records it would just be: cid.orig_p.port cid.orig_p.proto That doesn't seem to bad. -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Wed Nov 30 12:12:41 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 30 Nov 2011 20:12:41 -0000 Subject: [Bro-Dev] #651: Use "referer" rather than "referrer" in http.log In-Reply-To: <050.21e380b530abafbc75790d48a8544965@tracker.bro-ids.org> References: <050.21e380b530abafbc75790d48a8544965@tracker.bro-ids.org> Message-ID: <065.8fdf99d515fcddd0890cb760a8bc91c2@tracker.bro-ids.org> #651: Use "referer" rather than "referrer" in http.log -----------------------+------------------------ Reporter: matthias | Owner: seth Type: Problem | Status: closed Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: git/master Resolution: Rejected | Keywords: -----------------------+------------------------ Changes (by seth): * status: new => closed * resolution: => Rejected Comment: There doesn't seem to be much consensus on this name so I'm just going to close this ticket and leave the name as it is. Even though it may not be what someone expects at first since the protocol misspelled the word it should still be clear from the content. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Wed Nov 30 13:13:09 2011 From: seth at icir.org (Seth Hall) Date: Wed, 30 Nov 2011 16:13:09 -0500 Subject: [Bro-Dev] Call for opinions on logging framework syntax problem In-Reply-To: <4ED68816.6000806@icir.org> References: <40827F30-AF22-4993-885D-0A9C6B25C499@icsi.berkeley.edu> <4ED59D20.9050805@icir.org> <20111130163243.GE96269@icir.org> <4ED68816.6000806@icir.org> Message-ID: <0D291BCB-E411-4A22-96BA-6C1AD1C5DF86@icir.org> On Nov 30, 2011, at 2:46 PM, Gregor Maier wrote: > Yeah. I can see sets, vectors being a problem, however, wouldn't it > break in the same way as sets/vectors of record types? I don't think the logging framework supports sets or vectors of records. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20111130/b3e00881/attachment.bin From bernhard at ICSI.Berkeley.EDU Wed Nov 30 13:33:35 2011 From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann) Date: Wed, 30 Nov 2011 13:33:35 -0800 Subject: [Bro-Dev] Call for opinions on logging framework syntax problem In-Reply-To: <0D291BCB-E411-4A22-96BA-6C1AD1C5DF86@icir.org> References: <40827F30-AF22-4993-885D-0A9C6B25C499@icsi.berkeley.edu> <4ED59D20.9050805@icir.org> <20111130163243.GE96269@icir.org> <4ED68816.6000806@icir.org> <0D291BCB-E411-4A22-96BA-6C1AD1C5DF86@icir.org> Message-ID: <54AA1654-DC45-4EF1-B5E7-80C035BAC91D@ICSI.Berkeley.EDU> > >> Yeah. I can see sets, vectors being a problem, however, wouldn't it >> break in the same way as sets/vectors of record types? > > > I don't think the logging framework supports sets or vectors of records. That's right, it does not. One could simply choose to disable the logging of sets/vectors of ports and use different predefined column names - however this somehow does not feel right, since a port is a basic type. From bro at tracker.bro-ids.org Wed Nov 30 14:32:16 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 30 Nov 2011 22:32:16 -0000 Subject: [Bro-Dev] #658: Broctl check doesn't account for cluster layout or config changes In-Reply-To: <046.33069865e6009f5d564323acf1468ca8@tracker.bro-ids.org> References: <046.33069865e6009f5d564323acf1468ca8@tracker.bro-ids.org> Message-ID: <061.7c5786307a171057e8df5d2684fedc82@tracker.bro-ids.org> #658: Broctl check doesn't account for cluster layout or config changes -------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: Resolution: | Keywords: -------------------------+-------------------- Comment (by jsiwek): In [54d5bf7734e13f58a9c72fabfa2cd952101a27fe/bro]: {{{ #!CommitTicketReference repository="bro" revision="54d5bf7734e13f58a9c72fabfa2cd952101a27fe" Change to way cluster framework @loads local-.bro scripts. It's subtle, but it means the difference between BroControl finding the script in the original, $prefix/share/bro/site location versus the one copied inside the spool directory. Addresses #658 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 30 14:37:47 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 30 Nov 2011 22:37:47 -0000 Subject: [Bro-Dev] #658: Broctl check doesn't account for cluster layout or config changes In-Reply-To: <046.33069865e6009f5d564323acf1468ca8@tracker.bro-ids.org> References: <046.33069865e6009f5d564323acf1468ca8@tracker.bro-ids.org> Message-ID: <061.595ad469117470f0b4e8910cee0ef2dd@tracker.bro-ids.org> #658: Broctl check doesn't account for cluster layout or config changes -------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: Resolution: | Keywords: -------------------------+-------------------- Comment (by jsiwek): In [c0437348025fab91d75d4afa0075cf0b32e087e1/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="c0437348025fab91d75d4afa0075cf0b32e087e1" Teach 'check' command to generate temporary versions of autogen. files. This helps the 'check' command catch errors in configuration files before installing scripts that were auto-generated from them. Addresses #658. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 30 14:39:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 30 Nov 2011 22:39:50 -0000 Subject: [Bro-Dev] #658: Broctl check doesn't account for cluster layout or config changes In-Reply-To: <046.33069865e6009f5d564323acf1468ca8@tracker.bro-ids.org> References: <046.33069865e6009f5d564323acf1468ca8@tracker.bro-ids.org> Message-ID: <061.2a769be880c5615c8a3448bf07265d98@tracker.bro-ids.org> #658: Broctl check doesn't account for cluster layout or config changes ----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro2.0 Component: BroControl | Version: Resolution: | Keywords: ----------------------------+-------------------- Changes (by jsiwek): * type: Problem => Merge Request Comment: Fixes ready to merge in branches `topic/jsiwek/ticket658` in both `bro` and `broctl` repos. -- Ticket URL: Bro Tracker Bro Issue Tracker From vern at icir.org Wed Nov 30 14:46:48 2011 From: vern at icir.org (Vern Paxson) Date: Wed, 30 Nov 2011 14:46:48 -0800 Subject: [Bro-Dev] state of $BRO_LOG_SUFFIX ? Message-ID: <20111130224648.2DA782C4008@rock.ICSI.Berkeley.EDU> It looks like for the most part this is no longer used. Hoewver, I see at least two exceptions. The first is the function log_file_name() in util.cc, which appears to now only be used by DebugLogger.cc. The second is open_log_file() in base/init-bare.bro (which calls log_file_name()), which only appears to be called by policy/misc/profiling.bro. (I've held off on filing a ticket on this until I understand the view of what's expected.) Vern From vern at icir.org Wed Nov 30 15:03:00 2011 From: vern at icir.org (Vern Paxson) Date: Wed, 30 Nov 2011 15:03:00 -0800 Subject: [Bro-Dev] semantics of ts field for known_services? Message-ID: <20111130230300.9181A2C4008@rock.ICSI.Berkeley.EDU> Is this field meant to capture when the determination was made that a given service is running somewhere? For a slice-trace I'm analyzing, I see it's on the ACK by the client of the first line sent back by the server. Not quite what I would expect, but also not necessarily any sort of issue. Vern From bro at tracker.bro-ids.org Wed Nov 30 15:09:40 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 30 Nov 2011 23:09:40 -0000 Subject: [Bro-Dev] #703: NUL_in_line and line_terminated_with_single_CR complaints Message-ID: <046.a178d88fb4159c54e316119fd1f5ded7@tracker.bro-ids.org> #703: NUL_in_line and line_terminated_with_single_CR complaints ---------------------+----------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: Keywords: | ---------------------+----------------- It would be nice to suppress these for connections that negotiated encryption, such as START_TLS for SMTP, since in that context these occurrences are not at all anomalous. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Wed Nov 30 15:10:58 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 30 Nov 2011 15:10:58 -0800 Subject: [Bro-Dev] state of $BRO_LOG_SUFFIX ? In-Reply-To: <20111130224648.2DA782C4008@rock.ICSI.Berkeley.EDU> References: <20111130224648.2DA782C4008@rock.ICSI.Berkeley.EDU> Message-ID: <20111130231058.GF12839@icir.org> Yeah, that's essentially not supported anymore right now. As you write, open_log_file() will still apply it, but almost nothing goes through that anymore. I believe this should be easy to fix for the ASCII writer. It can see the environment variable and would just need to add it to the filenames it generates. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From vern at icir.org Wed Nov 30 15:24:25 2011 From: vern at icir.org (Vern Paxson) Date: Wed, 30 Nov 2011 15:24:25 -0800 Subject: [Bro-Dev] question & meta-question regarding "path" field in smtp.log Message-ID: <20111130232425.564422C4004@rock.ICSI.Berkeley.EDU> The question is: what's the semantics of this log field? In the trace slice I'm looking at, it's just the server,client. Is it meant to have the Received chain? The meta-question is: how am I supposed to find documentation for stuff like this? Searching on "smtp path" from bro-ids.org didn't bring up obvious hits. It wasn't that hard to navigate my way to: http://bro-ids.org/documentation-beta/bro-scripts/scripts/base/protocols/smtp/main.html but there all I find is a listing of the type of "path", but not its meaning: path: vector &log &optional Is this just because the documentation hasn't yet caught up this far? Vern From bro at tracker.bro-ids.org Wed Nov 30 15:42:38 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 30 Nov 2011 23:42:38 -0000 Subject: [Bro-Dev] #704: use of $BRO_LOG_SUFFIX Message-ID: <046.a58006125a50eea1b4c28ca8e55b2431@tracker.bro-ids.org> #704: use of $BRO_LOG_SUFFIX ---------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Keywords: | ---------------------+-------------------- This is now inconsistently used, per the following email exchange: Vern: It looks like for the most part this is no longer used. Hoewver, I see at least two exceptions. The first is the function log_file_name() in util.cc, which appears to now only be used by DebugLogger.cc. The second is open_log_file() in base/init-bare.bro (which calls log_file_name()), which only appears to be called by policy/misc/profiling.bro. Robin: Yeah, that's essentially not supported anymore right now. As you write, open_log_file() will still apply it, but almost nothing goes through that anymore. I believe this should be easy to fix for the ASCII writer. It can see the environment variable and would just need to add it to the filenames it generates. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Nov 30 15:43:17 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 30 Nov 2011 23:43:17 -0000 Subject: [Bro-Dev] #705: Makefile in doc/ doesn't work. Message-ID: <047.924679682a5a6c6c0784fa8db12eb41f@tracker.bro-ids.org> #705: Makefile in doc/ doesn't work. ---------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro2.0 Component: Bro | Version: Keywords: | ---------------------+-------------------- There's still the old Makefile in `doc/`, which doesn't work anymore. And the README instructs to use `make doc` or `make broxygen`, but neither works. -- Ticket URL: Bro Tracker Bro Issue Tracker From vern at icir.org Wed Nov 30 16:10:24 2011 From: vern at icir.org (Vern Paxson) Date: Wed, 30 Nov 2011 16:10:24 -0800 Subject: [Bro-Dev] buglet in FTP processing for 1.E7Vo2Cjyt39.pcap in slice 9 Message-ID: <20111201001024.7F85F2C4004@rock.ICSI.Berkeley.EDU> The FTP log file reports the size of the transferred items as 76 and 7 bytes, while in the dialog they're actually reported in KB (apparently): 150 Binary transfer started (76k). .... 150 Binary transfer started (7k). I'm not sure whether these are worth worrying about, though. Presumably if the accompanying ftp-data connections were also in the trace, the sizes would come from those (?). Vern From vern at icir.org Wed Nov 30 17:03:09 2011 From: vern at icir.org (Vern Paxson) Date: Wed, 30 Nov 2011 17:03:09 -0800 Subject: [Bro-Dev] generation of loaded_scripts.log Message-ID: <20111201010309.902C32C4007@rock.ICSI.Berkeley.EDU> Why does this occur if I run using "local tuning/track-all-assets", but not if I run with just "tuning/track-all-assets"? I don't understand why dumping out the loaded scripts only matters for "local", though maybe the notion is that with "local" it's hard to track just what might get loaded .... ? Vern From bro at tracker.bro-ids.org Wed Nov 30 19:57:10 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 01 Dec 2011 03:57:10 -0000 Subject: [Bro-Dev] #703: NUL_in_line and line_terminated_with_single_CR complaints In-Reply-To: <046.a178d88fb4159c54e316119fd1f5ded7@tracker.bro-ids.org> References: <046.a178d88fb4159c54e316119fd1f5ded7@tracker.bro-ids.org> Message-ID: <061.baf9c6dfa5b541fbe2f30bdc28ce00df@tracker.bro-ids.org> #703: NUL_in_line and line_terminated_with_single_CR complaints ----------------------+----------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: Resolution: | Keywords: ----------------------+----------------- Comment (by seth): > It would be nice to suppress these for connections that negotiated > encryption, such as START_TLS for SMTP, since in that context these > occurrences are not at all anomalous. I want to handle that correctly soon by passing the data to the SSL analyzer once the SSL upgrade occurs. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Wed Nov 30 20:06:39 2011 From: seth at icir.org (Seth Hall) Date: Wed, 30 Nov 2011 23:06:39 -0500 Subject: [Bro-Dev] semantics of ts field for known_services? In-Reply-To: <20111130230300.9181A2C4008@rock.ICSI.Berkeley.EDU> References: <20111130230300.9181A2C4008@rock.ICSI.Berkeley.EDU> Message-ID: On Nov 30, 2011, at 6:03 PM, Vern Paxson wrote: > Is this field meant to capture when the determination was made that a given > service is running somewhere? For a slice-trace I'm analyzing, I see it's > on the ACK by the client of the first line sent back by the server. Not > quite what I would expect, but also not necessarily any sort of issue. The semantics of that field are a little fuzzy. If a protocol was detected, the field contains the time that the analyzer generated the ProtocolConfirmation. If no protocol was detected, a scheduled event is set for several minutes (I think 5 by default) so that Bro can wait and see if a better connection where a protocol is detected comes along before it goes to log the service. Hm, I guess the semantics are pretty clear, the ts field always contains the time when the log record was written. Determining why that happened when it did is a bit fuzzy. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Wed Nov 30 20:07:24 2011 From: seth at icir.org (Seth Hall) Date: Wed, 30 Nov 2011 23:07:24 -0500 Subject: [Bro-Dev] Call for opinions on logging framework syntax problem In-Reply-To: <54AA1654-DC45-4EF1-B5E7-80C035BAC91D@ICSI.Berkeley.EDU> References: <40827F30-AF22-4993-885D-0A9C6B25C499@icsi.berkeley.edu> <4ED59D20.9050805@icir.org> <20111130163243.GE96269@icir.org> <4ED68816.6000806@icir.org> <0D291BCB-E411-4A22-96BA-6C1AD1C5DF86@icir.org> <54AA1654-DC45-4EF1-B5E7-80C035BAC91D@ICSI.Berkeley.EDU> Message-ID: <04C960E7-9B00-4802-A656-EAC98C1EAC94@icir.org> On Nov 30, 2011, at 4:33 PM, Bernhard Amann wrote: > One could simply choose to disable the logging of sets/vectors of ports and use different predefined column names - however this somehow does not feel right, since a port is a basic type. Agreed. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Wed Nov 30 20:08:22 2011 From: seth at icir.org (Seth Hall) Date: Wed, 30 Nov 2011 23:08:22 -0500 Subject: [Bro-Dev] question & meta-question regarding "path" field in smtp.log In-Reply-To: <20111130232425.564422C4004@rock.ICSI.Berkeley.EDU> References: <20111130232425.564422C4004@rock.ICSI.Berkeley.EDU> Message-ID: On Nov 30, 2011, at 6:24 PM, Vern Paxson wrote: > Is this just because the documentation hasn't yet caught up this far? Exactly. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Wed Nov 30 20:09:55 2011 From: seth at icir.org (Seth Hall) Date: Wed, 30 Nov 2011 23:09:55 -0500 Subject: [Bro-Dev] question & meta-question regarding "path" field in smtp.log In-Reply-To: <20111130232425.564422C4004@rock.ICSI.Berkeley.EDU> References: <20111130232425.564422C4004@rock.ICSI.Berkeley.EDU> Message-ID: <09B66D9F-B932-41F4-A482-31D971CBD0B5@icir.org> On Nov 30, 2011, at 6:24 PM, Vern Paxson wrote: > The question is: what's the semantics of this log field? In the > trace slice I'm looking at, it's just the server,client. Is it meant > to have the Received chain? I forgot to answer this question too. Yes, it's the path derived from the received headers and it's in reverse order where the actual message originator would be found at the right side and the receiver would be at the left. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Wed Nov 30 20:16:43 2011 From: seth at icir.org (Seth Hall) Date: Wed, 30 Nov 2011 23:16:43 -0500 Subject: [Bro-Dev] buglet in FTP processing for 1.E7Vo2Cjyt39.pcap in slice 9 In-Reply-To: <20111201001024.7F85F2C4004@rock.ICSI.Berkeley.EDU> References: <20111201001024.7F85F2C4004@rock.ICSI.Berkeley.EDU> Message-ID: On Nov 30, 2011, at 7:10 PM, Vern Paxson wrote: > 150 Binary transfer started (7k). Oops, I haven't seen FTP servers do that before. I'll fix it. > I'm not sure whether these are worth worrying about, though. Presumably > if the accompanying ftp-data connections were also in the trace, the > sizes would come from those (?). I don't think the ftp base scripts pull data from that currently. That would make sense though. :) One problem with it is that in cluster deployments it's very common for the data connections to not be analyzed appropriately (it only works if 2-tuple load balancing is taking place). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Wed Nov 30 20:18:00 2011 From: seth at icir.org (Seth Hall) Date: Wed, 30 Nov 2011 23:18:00 -0500 Subject: [Bro-Dev] generation of loaded_scripts.log In-Reply-To: <20111201010309.902C32C4007@rock.ICSI.Berkeley.EDU> References: <20111201010309.902C32C4007@rock.ICSI.Berkeley.EDU> Message-ID: On Nov 30, 2011, at 8:03 PM, Vern Paxson wrote: > Why does this occur if I run using "local tuning/track-all-assets", but > not if I run with just "tuning/track-all-assets"? I don't understand > why dumping out the loaded scripts only matters for "local", though > maybe the notion is that with "local" it's hard to track just what > might get loaded .... ? The local script is where we are suggesting a starting configuration for users, similarly to how Apache probably ships a default httpd.conf. The misc/loaded-scripts script is loaded in the local script. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/