From E.H.Beekman at amc.nl Mon Mar 1 06:34:33 2010 From: E.H.Beekman at amc.nl (Ewald Beekman) Date: Mon, 1 Mar 2010 15:34:33 +0100 Subject: [Bro] BRO & Malware Hash Registry Message-ID: <20100301143433.GS22986@core2.amc.nl> Hi, I would like to run Bro 1.5.1. with lookups to the MHR, since the 1.5 code allready contains the MD5 functions (?), i assumed i only needed the: http-cymru-malware-hash.bro which i loaded into my site/local.bro @load http-cymru-malware-hash.bro and verified to be present through: # broctl scripts | & grep cymru http-cymru-malware-hash.bro But i don't see any related logging, so i'm afraid of being a little naive in my approach. Since i am totally new at bro, i don't have a clue at how to debug the script. thanks in advance for any help. Ewald... BTW i really like the functionality in Bro, it helped me discover bot IRC traffic on port 80 and inst_n105.exe trojan dropper downloads from a server in Russia. From hall.692 at osu.edu Mon Mar 1 06:52:18 2010 From: hall.692 at osu.edu (Seth Hall) Date: Mon, 1 Mar 2010 09:52:18 -0500 Subject: [Bro] BRO & Malware Hash Registry In-Reply-To: <20100301143433.GS22986@core2.amc.nl> References: <20100301143433.GS22986@core2.amc.nl> Message-ID: On Mar 1, 2010, at 9:34 AM, Ewald Beekman wrote: > I would like to run Bro 1.5.1. with lookups to the MHR, > since the 1.5 code allready contains the MD5 functions (?), > i assumed i only needed the: Sorry about that. The situation with that script has kind of been a mess for a while. Recently I spent some time reworking it and integrating it into the rest of my scripts better. I'll finish that up today and update the repository. It will be much easier to run the script once I update it. The logging will be a lot better too. I'll send an email when I make the update. .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From govind at ac.upc.edu Mon Mar 1 12:50:50 2010 From: govind at ac.upc.edu (Govind) Date: Mon, 01 Mar 2010 21:50:50 +0100 Subject: [Bro] regarding regular expression matching Message-ID: <1267476650.11986.44.camel@vinayaka> Greetings all, I am trying to get some statistics on reg-exp matching in IDS, for that I am using Bro. Further, I am also using Snort signatures (converted using s2b utility) as input. I have a few questions and need a clarification as well on the Bro code. The way Bro does reg-exp matching is by building a tree of rules (not more than RE_level I guess). Looking at the RuleMatcher.cc code, the way it is done is by grouping signatures. A set of signatures are then compiled and a DFA is built for each set and the process repated for the entire tree. Have I got it correctly ?. However, what I do not completely understand is what is the condition for a node to be a child of another node. Dumping the tree structure out, I see that a node (a particular signature) with same src-port range and ip protocol is at one level while another node (another signature) (with same configs) is at a lower level (down the tree). What exactly is the importance of RE_level, is it to have any caching advantages ? Additionally, each DFA machine (a DFA built using one set of reg-exps) has its own state cache. A single centralised cache is not used. Is it right (I disable EXPIRE_DFA_STATE flag) ? Finally, I see that the in-built Bro rules are built separately. While building the DFA for this, I notice that 2 copies of the same regular-expression is built - one called MATCH ANYWHERE and other MATCH EXACTLY. In MATCH EXACTLY, I see that the same reg-exp as used in MATCH ANYWHERE is called but with the restriction that the reg-exp has to start at beginning of line. Would'nt MATCH EXACTLY be a subset of MATCH ANYWHERE ?. Finally, I do not see MATCH EXACTLY and MATCH ANYWHERE being done for snort imported signatures (I am using only a small subset currently). Is it true for all snort converted signatures ? Thank you for your very patient reading! Regards Govind From addyg420 at yahoo.com Tue Mar 2 06:10:52 2010 From: addyg420 at yahoo.com (add gy) Date: Tue, 2 Mar 2010 06:10:52 -0800 (PST) Subject: [Bro] bro - broccoli - gcc error Message-ID: <357522.78301.qm@web45305.mail.sp1.yahoo.com> Please suggest on below error ? broccoli_intern_wrap.c:8558: warning: implicit declaration of function ?SWIG_Python_SetConstant? broccoli_intern_wrap.c:8558: warning: implicit declaration of function ?SWIG_From_int? error: command 'gcc' failed with exit status 1 make[4]: *** [pybroccoli] Error 1 make[4]: Leaving directory `/tmp/bro-1.5.1/aux/broctl' make[3]: *** [all-recursive] Error 1 make[3]: Leaving directory `/tmp/bro-1.5.1/aux' make[2]: *** [all] Error 2 make[2]: Leaving directory `/tmp/bro-1.5.1/aux' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/tmp/bro-1.5.1' make: *** [all] Error 2 ? ./configure is successfuly completed error occures when i run make . ? Regards, Add Get your new Email address! Grab the Email name you've always wanted before someone else does! http://mail.promotions.yahoo.com/newdomains/aa/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20100302/10bc389b/attachment.html From robin at icir.org Tue Mar 2 09:30:33 2010 From: robin at icir.org (Robin Sommer) Date: Tue, 2 Mar 2010 09:30:33 -0800 Subject: [Bro] bro - broccoli - gcc error In-Reply-To: <357522.78301.qm@web45305.mail.sp1.yahoo.com> References: <357522.78301.qm@web45305.mail.sp1.yahoo.com> Message-ID: <20100302173033.GB48874@icir.org> On Tue, Mar 02, 2010 at 06:10 -0800, add gy wrote: > broccoli_intern_wrap.c:8558: warning: implicit declaration of function ?SWIG_Python_SetConstant? > broccoli_intern_wrap.c:8558: warning: implicit declaration of function ?SWIG_From_int? > error: command 'gcc' failed with exit status 1 The messages about broccoli are only warnings so not sure what's actually triggering the error. But try this patch: http://tracker.icir.org/bro/ticket/238 ... and see if it helps. Robin -- Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From JAzoff at uamail.albany.edu Tue Mar 2 10:18:59 2010 From: JAzoff at uamail.albany.edu (Justin Azoff) Date: Tue, 2 Mar 2010 13:18:59 -0500 Subject: [Bro] bro - broccoli - gcc error In-Reply-To: <357522.78301.qm@web45305.mail.sp1.yahoo.com> References: <357522.78301.qm@web45305.mail.sp1.yahoo.com> Message-ID: <20100302181858.GI26162@datacomm.albany.edu> On Tue, Mar 02, 2010 at 06:10:52AM -0800, add gy wrote: > Please suggest on below error > > broccoli_intern_wrap.c:8558: warning: implicit declaration of function ?SWIG_Python_SetConstant? > broccoli_intern_wrap.c:8558: warning: implicit declaration of function ?SWIG_From_int? > error: command 'gcc' failed with exit status 1 > make[4]: *** [pybroccoli] Error 1 I think this is the error that you get when you are missing the python development package, which is usually called something like python-dev. -- -- Justin Azoff -- Network Security & Performance Analyst From powellsm at musc.edu Tue Mar 2 13:06:53 2010 From: powellsm at musc.edu (Powell, Scott) Date: Tue, 2 Mar 2010 16:06:53 -0500 Subject: [Bro] Timemachine and Bro 1.5.1? Message-ID: I just downloaded and configured timemachine (http://www.net.t-labs.tu-berlin.de/research/tm/#download) and am attempting to integrate it with Bro 1.5.1. Upon defining my timemachinehost in broctl.conf and restarting Bro it caused my timemachine process to abort with the following error: *** Broccoli error: bro_conn_new_socket called without prior initialization. *** Initialization of the Broccoli library is now required. *** See documentation for details. Aborting. I noticed the latest download of timemachine is from February of 2009. However Bro 1.5.1 is a lot newer than that. Is anyone running the newest Bro IDS with timemachine? If so what is the workaround or fix for the above error? Thanks, Scott Powell Unix Systems Engineer / Information Security Analyst Office of the CIO - Information Systems (OCIO-IS) Medical University of South Carolina powellsm at musc.edu (843) 792-6651 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20100302/c3dcb920/attachment.html From hall.692 at osu.edu Tue Mar 2 13:28:48 2010 From: hall.692 at osu.edu (Seth Hall) Date: Tue, 2 Mar 2010 16:28:48 -0500 Subject: [Bro] Timemachine and Bro 1.5.1? In-Reply-To: References: Message-ID: On Mar 2, 2010, at 4:06 PM, Powell, Scott wrote: > I noticed the latest download of timemachine is from February of > 2009. However Bro 1.5.1 is a lot newer than that. Is anyone running > the newest Bro IDS with timemachine? If so what is the workaround or > fix for the above error? The Broccoli API changed and Time Machine was not updated for the change. I suppose this is as good of a time as any to announce the move of the Time Machine project from TU-Berlin to ICSI. The source code repository is now public and there is a ticket tracker. You should be able to find everything at the following URL: http://tracker.icir.org/time-machine The change needed is actually very small and someone may provide a patch, but I'm hoping in the next day or two to get some time (sigh) to do the patch and make a new release of Time Machine available. I'd also like to say thanks to all involved parties for making it possible to move the Time Machine project to ICSI! Thanks, .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From JAzoff at uamail.albany.edu Tue Mar 2 13:49:41 2010 From: JAzoff at uamail.albany.edu (Justin Azoff) Date: Tue, 2 Mar 2010 16:49:41 -0500 Subject: [Bro] Timemachine and Bro 1.5.1? In-Reply-To: References: Message-ID: <20100302214940.GJ26162@datacomm.albany.edu> On Tue, Mar 02, 2010 at 04:28:48PM -0500, Seth Hall wrote: > > On Mar 2, 2010, at 4:06 PM, Powell, Scott wrote: > > > I noticed the latest download of timemachine is from February of > > 2009. However Bro 1.5.1 is a lot newer than that. Is anyone running > > the newest Bro IDS with timemachine? If so what is the workaround or > > fix for the above error? > The change needed is actually very small and someone may provide a > patch, but I'm hoping in the next day or two to get some time (sigh) > to do the patch and make a new release of Time Machine available. See attached, it works for me :-) -- -- Justin Azoff -- Network Security & Performance Analyst -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Initialize-broccoli-library.patch Type: text/x-diff Size: 846 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20100302/7d2336da/attachment.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20100302/7d2336da/attachment-0001.bin From addyg420 at yahoo.com Tue Mar 2 23:41:28 2010 From: addyg420 at yahoo.com (add gy) Date: Tue, 2 Mar 2010 23:41:28 -0800 (PST) Subject: [Bro] bro - broccoli - gcc error Message-ID: <29138.1099.qm@web45315.mail.sp1.yahoo.com> Thanks ? Justin Azoff & Robin Sommer ? Installing python-dev solved my problem ? Regards, Add Get your preferred Email name! Now you can @ymail.com and @rocketmail.com. http://mail.promotions.yahoo.com/newdomains/aa/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20100302/f90c7c90/attachment.html From mccreary at ucar.edu Thu Mar 4 16:38:58 2010 From: mccreary at ucar.edu (Sean McCreary) Date: Thu, 04 Mar 2010 17:38:58 -0700 Subject: [Bro] internal error: unknown msg type 101 in Poll() In-Reply-To: <21398958-20C5-418D-94AC-9A4B3CA96CD4@osu.edu> References: <4B7FFD0F.2060403@ucar.edu> <21398958-20C5-418D-94AC-9A4B3CA96CD4@osu.edu> Message-ID: <4B9052A2.10905@ucar.edu> On 20/02/10 11:48, Seth Hall wrote: > On Feb 20, 2010, at 10:17 AM, Sean McCreary wrote: > >> I have been seeing several crashes per day due to 'internal error: >> unknown msg type 101 in Poll()' in the manager process of a bro >> cluster >> handling ~2.5 Gb/s of traffic. Here is a typical stack trace: > > > Try two things. [...] > 2. Add the following to your local.bro script: > redef notice_action_filters += { > [Weird::ContentGap] = ignore_notice, > [Weird::AckAboveHole] = ignore_notice, > }; > redef suppress_notice_actions += { > Weird::ContentGap, > Weird::AckAboveHole, > }; FYI, this policy change was sufficient to fix the stability problems in my cluster. I now believe the underlying problem is upstream packet loss, and I have been working to eliminate that problem in my traffic capture and distribution network. In hindsight, I've had similar problems in the past with overloaded standalone Bro processes. Missing packets generate a significant amount of additional load, and this tends to turn a small problem into a more serious one. Perhaps we can start a 'debugging Bro problems' page on the wiki, or add this policy to an existing one? From robin at icir.org Fri Mar 5 08:43:04 2010 From: robin at icir.org (Robin Sommer) Date: Fri, 5 Mar 2010 08:43:04 -0800 Subject: [Bro] internal error: unknown msg type 101 in Poll() In-Reply-To: <4B9052A2.10905@ucar.edu> References: <4B7FFD0F.2060403@ucar.edu> <21398958-20C5-418D-94AC-9A4B3CA96CD4@osu.edu> <4B9052A2.10905@ucar.edu> Message-ID: <20100305164304.GD1037@icir.org> Can somebody please file a ticket with the recommendations collected in this thread? I'll see what makes sense to integrate into the default cluster config. (The Wiki page is still a good idea, and it's fine for the ticket to just link there.) Thanks, Robin -- Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From hall.692 at osu.edu Fri Mar 5 09:18:01 2010 From: hall.692 at osu.edu (Seth Hall) Date: Fri, 5 Mar 2010 12:18:01 -0500 Subject: [Bro] internal error: unknown msg type 101 in Poll() In-Reply-To: <20100305164304.GD1037@icir.org> References: <4B7FFD0F.2060403@ucar.edu> <21398958-20C5-418D-94AC-9A4B3CA96CD4@osu.edu> <4B9052A2.10905@ucar.edu> <20100305164304.GD1037@icir.org> Message-ID: On Mar 5, 2010, at 11:43 AM, Robin Sommer wrote: > Can somebody please file a ticket with the recommendations collected > in this thread? I'll see what makes sense to integrate into the > default cluster config. Done. http://tracker.icir.org/bro/ticket/245 .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From E.H.Beekman at amc.nl Wed Mar 10 02:29:53 2010 From: E.H.Beekman at amc.nl (Ewald Beekman) Date: Wed, 10 Mar 2010 11:29:53 +0100 Subject: [Bro] BRO & Malware Hash Registry In-Reply-To: References: <20100301143433.GS22986@core2.amc.nl> Message-ID: <20100310102953.GM2913@blackbeauty.local> Hi Seth, Sorry for bothering you, but .. did you get any further with the cleanup of the code? I have used Bro for about two weeks now and it helps me detect malware infections very well (i check for suspicious downloads and redownload them myself). Allthough i really would like the automation to check agains the MHR, my personal findings indicate that the hashes are commonly not know. Would it be an idea to check the hash against virustotal? http://www.virustotal.com/buscaHash.html don't know if the scripting language of Bro is even capable of doing that. I wrote a perl script to do it from the commandline but it uses WWW::Mechanize. Perhaps a better idea for BRO/VT capabilities is to use an intermediate system which does the hash checking with VT and caches the results. Bro could than use simple http to check the hash against the intermediate system. just my thoughts, free flowing ;) thanks in advance, Ewald Beekman... On Mon, Mar 01, 2010 at 09:52:18AM -0500, Seth Hall wrote: > > On Mar 1, 2010, at 9:34 AM, Ewald Beekman wrote: > >> I would like to run Bro 1.5.1. with lookups to the MHR, >> since the 1.5 code allready contains the MD5 functions (?), >> i assumed i only needed the: > > > Sorry about that. The situation with that script has kind of been a > mess for a while. Recently I spent some time reworking it and > integrating it into the rest of my scripts better. I'll finish that up > today and update the repository. It will be much easier to run the > script once I update it. The logging will be a lot better too. > > I'll send an email when I make the update. > > .Seth > > --- > Seth Hall > Network Security - Office of the CIO > The Ohio State University > Phone: 614-292-9721 > -- Ewald Beekman, CISSP. Academic Medical Center (AMC), NL From estrada.veronica at gmail.com Wed Mar 10 06:46:06 2010 From: estrada.veronica at gmail.com (Veronica Estrada) Date: Wed, 10 Mar 2010 23:46:06 +0900 Subject: [Bro] processing many files with bro Message-ID: Hello, I am processing several hours of captured traffic split into pcap files that covers 1 minute traffic each. Actually I am having this basic script to do that. #!/bin/bash path=("$@") for f in $(ls $path);do export BRO_LOG_SUFFIX=$f; /usr/local/bro/bin/bro -r $path/$f brolite mysite done But my goal is that bro recognize connections that could be split in several files. I am thinking that one solution is to modified some variables and make them "persistent". Is it correct? Which variables should I modified? The other solution. I know that split pcap files can be merged in one bigger file, but I will have problems with memory, and bro may crash if it has a limitation for processing big size pcap file. So I am not considering this option. Best regards! Veronica Estrada Nakao Laboratory The University of Tokyo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20100310/5c811b8a/attachment.html From hall.692 at osu.edu Wed Mar 10 08:04:06 2010 From: hall.692 at osu.edu (Seth Hall) Date: Wed, 10 Mar 2010 11:04:06 -0500 Subject: [Bro] BRO & Malware Hash Registry In-Reply-To: <20100310102953.GM2913@blackbeauty.local> References: <20100301143433.GS22986@core2.amc.nl> <20100310102953.GM2913@blackbeauty.local> Message-ID: <34A4EBD9-6CE6-4DC7-B2C9-906689AE8E62@osu.edu> On Mar 10, 2010, at 5:29 AM, Ewald Beekman wrote: > Sorry for bothering you, but .. > did you get any further with the cleanup of the code? I did. :) On Monday I pushed my changes to my my github repository but I just haven't had time to send an email about the changes yet. There are a lot of changes. * My version of http-identified-files is now named http-ext-identified- files to avoid conflicts with the built in http-identified-files script. * http-ext-identified-files doesn't require libmagic anymore and file signatures are now defined in http-ext-identified-files.sig. * Software-ext can detect and log adobe flash player versions (more detected software coming soon) if http-ext is loaded. - you also want to apply this patch if you load software-ext.. http://tracker.icir.org/bro/ticket/227 * Team Cymru MHR work is now done by http-hash.bro and the notice name has changed from HTTP_Malware to HTTP_MHR_Malware. This could change again at some point, I can't figure out which I like better. * Please don't run dns-ext.bro. It *will* cause memory issues because it doesn't release state quickly enough. I'm going to fix that problem as soon as I have time and motivation. If you want to see configuration options for the scripts, look in the "export" section near the top. I think I've documented each of the configuration options, but let me know if there are anymore options you'd like to have. I'm going to be writing documentation for all of these scripts soon with more detail than just "look at the export section". There are a lot of little tricks you can do to do further analysis that might not be readily apparent. > I have used Bro for about two weeks now and it helps me > detect malware infections very well Great! The new version of the scripts makes it much easier to define what types of files you'd like to collect hashes for and logs those in the normal tab separated output format that I use for all of my scripts (documented in http-hash). Keep in mind that your Bro instance will slow down as you generate md5 sums for more file types. It could be worth testing the limits on your network though if you're interested in md5 sums for a few extra file types. > Would it be an idea to check the hash against virustotal? That's definitely a legitimate idea and it would be great if they offered a DNS interface similar to Team Cymru. However, they don't so we can't do it right now. > Perhaps a better idea for BRO/VT capabilities is to use an > intermediate system which does the hash checking with VT and > caches the results. Bro could than use simple http to check > the hash against the intermediate system. Matthias Vallentin has an idea for handling this sort of extended processing that can't currently be done (and possibly shouldn't be done) within Bro. I'll let him introduce his thoughts relating to your idea if he wants. > just my thoughts, free flowing ;) Keep them coming. :) .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From hall.692 at osu.edu Wed Mar 10 08:19:55 2010 From: hall.692 at osu.edu (Seth Hall) Date: Wed, 10 Mar 2010 11:19:55 -0500 Subject: [Bro] processing many files with bro In-Reply-To: References: Message-ID: <02C56686-EDB1-4FEF-93C8-FA559A2D1E74@osu.edu> On Mar 10, 2010, at 9:46 AM, Veronica Estrada wrote: > The other solution. I know that split pcap files can be merged in > one bigger file, but I will have problems with memory, and bro may > crash if it has a limitation for processing big size pcap file. So I > am not considering this option. I would go for this option. Bro *shouldn't* have memory problems as long as you are expiring all of the state that is accumulated often enough. When you run against the large tracefile, make sure you load the "profiling" script so you can see how much memory your various global variables are holding, that should tease out any variables which you may need to tune to reduce memory usage. Personally, I've processed a single multi-hundred gig tracefile with a single Bro instance on a machine with 512 megs of memory and didn't encounter any trouble. .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From hall.692 at osu.edu Wed Mar 10 08:20:43 2010 From: hall.692 at osu.edu (Seth Hall) Date: Wed, 10 Mar 2010 11:20:43 -0500 Subject: [Bro] BRO & Malware Hash Registry In-Reply-To: <34A4EBD9-6CE6-4DC7-B2C9-906689AE8E62@osu.edu> References: <20100301143433.GS22986@core2.amc.nl> <20100310102953.GM2913@blackbeauty.local> <34A4EBD9-6CE6-4DC7-B2C9-906689AE8E62@osu.edu> Message-ID: <00AB4364-795E-46ED-A7B3-BB143436AF95@osu.edu> On Mar 10, 2010, at 11:04 AM, Seth Hall wrote: > I did. :) On Monday I pushed my changes to my my github repository In case anyone doesn't know the URL: http://github.com/sethhall/bro_scripts .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From fabian at net.t-labs.tu-berlin.de Wed Mar 10 08:30:44 2010 From: fabian at net.t-labs.tu-berlin.de (Fabian Schneider) Date: Wed, 10 Mar 2010 17:30:44 +0100 Subject: [Bro] processing many files with bro In-Reply-To: References: Message-ID: <4B97C934.8080008@net.t-labs.tu-berlin.de> Veronica Estrada wrote: > Hello, > > I am processing several hours of captured traffic split into pcap files > that covers 1 minute traffic each. Actually I am having this basic > script to do that. > > #!/bin/bash > path=("$@") > for f in $(ls $path);do > export BRO_LOG_SUFFIX=$f; > /usr/local/bro/bin/bro -r $path/$f brolite mysite > done > > But my goal is that bro recognize connections that could be split in > several files. I am thinking that one solution is to modified some > variables and make them "persistent". Is it correct? Which variables > should I modified? I would rather try to write a pcap application that removes the pcap file headers from a set of input, and that have this application read the files one by one and pipe the output to bro. > The other solution. I know that split pcap files can be merged in one > bigger file, but I will have problems with memory, and bro may crash if > it has a limitation for processing big size pcap file. So I am not > considering this option. I am not aware of any problems of Bro reading huge input file. We are operationally using Bro and have instances analyze Terabytes of traces in one run. But of course the more data you put in the more state might be built up. best Fabian -- Fabian Schneider, An-Institut Deutsche Telekom Laboratories Technische Universitaet Berlin, Fakultaet IV -- Elektrotechnik und Informatik address: Sekr. TEL 4, FG INET, Ernst-Reuter-Platz 7, 10587 Berlin e-mail: fabian at net.t-labs.tu-berlin.de, WWW: http://www.net.t-labs.tu-berlin.de/~fabian phone: +49 30 8353 - 58513, mobile: +49 160 479 43 97 From robin at icir.org Wed Mar 10 08:30:56 2010 From: robin at icir.org (Robin Sommer) Date: Wed, 10 Mar 2010 08:30:56 -0800 Subject: [Bro] processing many files with bro In-Reply-To: References: Message-ID: <20100310163056.GJ62537@icir.org> On Wed, Mar 10, 2010 at 23:46 +0900, Veronica Estrada wrote: > The other solution. I know that split pcap files can be merged in one bigger > file, but I will have problems with memory, and bro may crash if it has a > limitation for processing big size pcap file. That's probably the best solution and you can do it on the fly: have your merge tool (e.g., tcpslice) write to stdout and Bro read from stdin with "-r -". The effect on memory will indeed be that of one large pcap file but if that causes trouble, you should to tweak the Bro configuration. Using &persistent is unlikely to do what you want as it stores only script-level state, not internal state for connections that cross file boundaries. Robin -- Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From estrada.veronica at gmail.com Wed Mar 10 09:15:29 2010 From: estrada.veronica at gmail.com (Veronica Estrada) Date: Thu, 11 Mar 2010 02:15:29 +0900 Subject: [Bro] processing many files with bro In-Reply-To: <20100310163056.GJ62537@icir.org> References: <20100310163056.GJ62537@icir.org> Message-ID: Thanks everyone for the answers, My original question was connected with a second problem. I am trying to associate a summary of wrong fragments to the corresponding line in the connection summary. To avoid the same connection becoming split and analyzed in different bro runs, I will go for second option as you suggested me. After that, I will have the majority of connections summarize in the same conn.bro file. But after solving this, I am still confused about how to associate the wrong fragment count with its corresponding connection logged in conn.bro To my understand, wrong fragments are generated in the flow_weird event and they don?t have associated a c$id, only src and dst address. My questions: 1. How can I check the connection that generated that wrong fragment event? 2. Should I assign the fragment to the last connection registered in the conn.bro who has connection initiation time before the fragment I want to count? I don?t think this is enough. For instance, if two different connections between A-B are active I cannot distinguish them. Besides, I read about active and pasive timeouts on connections (Flow-based TCP Connection Analysis by Limmer and Dressler). I don?t understand how this topic is treated in BRO. Since I can only find only one type of timeout (tcp_inactivity_timeout). Is this timeout the active timeout? I think probably there are others timeout such as handshake timeouts that I am missing. Maybe I am getting into the details of bro design, I want to understand what I am doing, and what I shouldn?t do to get the wrong fragment count inside the conn.bro file. Sorry, maybe I should open another thread with this e-mail. I was not sure how to deal with it. Veronica Estrada Nakao Laboratory The University of Tokyo On Thu, Mar 11, 2010 at 1:30 AM, Robin Sommer wrote: > > On Wed, Mar 10, 2010 at 23:46 +0900, Veronica Estrada wrote: > > > The other solution. I know that split pcap files can be merged in one > bigger > > file, but I will have problems with memory, and bro may crash if it has a > > limitation for processing big size pcap file. > > That's probably the best solution and you can do it on the fly: have > your merge tool (e.g., tcpslice) write to stdout and Bro read from > stdin with "-r -". The effect on memory will indeed be that of one > large pcap file but if that causes trouble, you should to tweak the > Bro configuration. > > Using &persistent is unlikely to do what you want as it stores only > script-level state, not internal state for connections that cross > file boundaries. > > Robin > > -- > Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org > ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20100311/caedd029/attachment.html From vallentin at ICSI.Berkeley.EDU Wed Mar 10 11:21:38 2010 From: vallentin at ICSI.Berkeley.EDU (Matthias Vallentin) Date: Wed, 10 Mar 2010 11:21:38 -0800 Subject: [Bro] processing many files with bro In-Reply-To: <20100310163056.GJ62537@icir.org> References: <20100310163056.GJ62537@icir.org> Message-ID: <20100310192138.GL629@icsi.berkeley.edu> On Wed, Mar 10, 2010 at 08:30:56AM -0800, Robin Sommer wrote: > That's probably the best solution and you can do it on the fly: have > your merge tool (e.g., tcpslice) write to stdout and Bro read from > stdin with "-r -". The effect on memory will indeed be that of one > large pcap file but if that causes trouble, you should to tweak the > Bro configuration. Yet another tool: % ipsumdump --collate -w - *.pcap | bro -r - http-request etc The switch --collate ensures monotone timestamps. Matthias -- Matthias Vallentin vallentin at icir.org http://www.icir.org/matthias From vallentin at ICSI.Berkeley.EDU Wed Mar 10 11:22:18 2010 From: vallentin at ICSI.Berkeley.EDU (Matthias Vallentin) Date: Wed, 10 Mar 2010 11:22:18 -0800 Subject: [Bro] BRO & Malware Hash Registry In-Reply-To: <34A4EBD9-6CE6-4DC7-B2C9-906689AE8E62@osu.edu> References: <20100301143433.GS22986@core2.amc.nl> <20100310102953.GM2913@blackbeauty.local> <34A4EBD9-6CE6-4DC7-B2C9-906689AE8E62@osu.edu> Message-ID: <20100310192218.GM629@icsi.berkeley.edu> > > Perhaps a better idea for BRO/VT capabilities is to use an > > intermediate system which does the hash checking with VT and > > caches the results. Bro could than use simple http to check > > the hash against the intermediate system. > > Matthias Vallentin has an idea for handling this sort of extended > processing that can't currently be done (and possibly shouldn't be > done) within Bro. I'll let him introduce his thoughts relating to > your idea if he wants. It is already possible to process Bro events from a scripting language (Ruby and Python currently) to perform time-intensive tasks separately, without having to worry about real-time constraints. My idea is to push this notion a little further by writing a framework that allows you to (i) manage intelligence sources in a unified fashion, e.g., blacklist integration (ii) generate/update both scripts and state remotely via broctl and the event-based Broccoli channel (iii) write high-level plug-ins (such as for Tor traffic, PDF analysis, or CWSandbox malware execution) that offer a consistent and interface to Bro using primitives from (ii) Seth brought point (i) to my attention, so I throw the ball back to him for details :-) Unfortunately, I am currently lacking the cycles to work on this idea. But as soon as any of this is usable, you'll hear about it. Matthias -- Matthias Vallentin vallentin at icir.org http://www.icir.org/matthias From vern at icir.org Wed Mar 10 11:30:31 2010 From: vern at icir.org (Vern Paxson) Date: Wed, 10 Mar 2010 11:30:31 -0800 Subject: [Bro] processing many files with bro In-Reply-To: <20100310192138.GL629@icsi.berkeley.edu> (Wed, 10 Mar 2010 11:21:38 PST). Message-ID: <201003101930.o2AJUVBv009033@pork.ICSI.Berkeley.EDU> > Yet another tool: > > % ipsumdump --collate -w - *.pcap | bro -r - http-request etc > > The switch --collate ensures monotone timestamps. Yeah, indeed that's a bit better than tcpslice, because ipsumdump will correctly collate traces that overlap in time, while IIRC tcpslice won't. Vern From estrada.veronica at gmail.com Thu Mar 11 03:01:47 2010 From: estrada.veronica at gmail.com (Veronica Estrada) Date: Thu, 11 Mar 2010 20:01:47 +0900 Subject: [Bro] processing many files with bro In-Reply-To: <20100310181457.GJ629@icsi.berkeley.edu> References: <20100310163056.GJ62537@icir.org> <20100310181457.GJ629@icsi.berkeley.edu> Message-ID: Sorry, I couldn't make it work. ipsumdump --collate -w *.pcap | $BROHOME/bin/bro -r - brolite mysite /usr/local/bro-1.5-dep/bin/bro: problem with trace file - - truncated dump file; tried to read 24 file header bytes, only got 0 Veronica On Thu, Mar 11, 2010 at 3:14 AM, Matthias Vallentin wrote: > On Wed, Mar 10, 2010 at 08:30:56AM -0800, Robin Sommer wrote: > > That's probably the best solution and you can do it on the fly: have > > your merge tool (e.g., tcpslice) write to stdout and Bro read from > > stdin with "-r -". The effect on memory will indeed be that of one > > large pcap file but if that causes trouble, you should to tweak the > > Bro configuration. > > Yet another tool: > > % ipsumdump --collate -w - *.pcap | bro -r - http-request etc > > The switch --collate ensures monotone timestamps. > > Matthias > -- > Matthias Vallentin > vallentin at icir.org > http://www.icir.org/matthias > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20100311/5c90ca57/attachment.html From estrada.veronica at gmail.com Thu Mar 11 03:22:54 2010 From: estrada.veronica at gmail.com (Veronica Estrada) Date: Thu, 11 Mar 2010 20:22:54 +0900 Subject: [Bro] linking a wrong_fragment event to a connection Message-ID: Hello everyone, I ask this topic again trying to clarify my questions (and my English). I want to associate a summary of wrong fragments to the corresponding line in the connection summary. I made a script to count the different fragment problems trigger by flow_weird event. How can I know which connection has generated that wrong fragment event? The wrong fragment event only logs src, dst and network_time. This is not enough to link the fragment to a connection inside connection summary. 1247652196.907274 src_ip -> dst_ip: fragment_with_DF By the way, I read about active and passive timeouts on connections ("Flow-based TCP Connection Analysis" by Limmer and Dressler). I don?t understand how this topic is treated in BRO. I found only one type of timeout (TCP_inactivity_timeout). Is this timeout the active timeout? Can I tune a passive timeout? Maybe I am missing others user tunable timeouts that can affect my results. Maybe I am getting into the details of bro design, I want to understand what I am doing, and what I shouldn?t do to get the wrong fragment count inside the conn.bro file. Veronica Estrada Nakao Laboratory The University of Tokyo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20100311/0d75ded2/attachment.html From vern at icir.org Thu Mar 11 07:31:30 2010 From: vern at icir.org (Vern Paxson) Date: Thu, 11 Mar 2010 07:31:30 -0800 Subject: [Bro] processing many files with bro In-Reply-To: (Thu, 11 Mar 2010 20:01:47 +0900). Message-ID: <201003111531.o2BFVUNW005897@pork.ICSI.Berkeley.EDU> > ipsumdump --collate -w *.pcap | $BROHOME/bin/bro -r - brolite mysite That needs to be: ipsumdump --collate -w - *.pcap | $BROHOME/bin/bro -r - brolite mysite Unfortunately as you issued it, ipsumdump wrote the output to the first file in the expansion of *.pcap, overwriting it :-(. Vern From vern at icir.org Thu Mar 11 07:56:48 2010 From: vern at icir.org (Vern Paxson) Date: Thu, 11 Mar 2010 07:56:48 -0800 Subject: [Bro] linking a wrong_fragment event to a connection In-Reply-To: (Thu, 11 Mar 2010 20:22:54 +0900). Message-ID: <201003111556.o2BFumCr006334@pork.ICSI.Berkeley.EDU> > How can I know which connection has generated that wrong fragment event? In general you can't. Most individual fragments do not possess transport headers, so there is no well-defined connection associated with them. *If* the fragment is part of a whole collection, then you can locate the transport header at the beginning of the collection and use its ports - but Bro may not have even seen this first part yet. So it only reports the involved hosts. (It also could in principle report ports if the problematic fragment happens to be the first *and* includes a full transport header [which it doesn't have to], but Bro doesn't go out of its way to do the extra work in this case.) > By the way, I read about active and passive timeouts on connections > ("Flow-based TCP Connection Analysis" by Limmer and Dressler). > I don't understand how this topic is treated in BRO. You'll need to explain how those timeouts are defined in that paper for others to be able to comment on how Bro's connection timeouts relate to them. Vern From mathew.binkley at vanderbilt.edu Thu Mar 11 11:01:36 2010 From: mathew.binkley at vanderbilt.edu (Mathew Binkley) Date: Thu, 11 Mar 2010 13:01:36 -0600 Subject: [Bro] Proper syntax for ignoring subnet to subnet traffic In-Reply-To: <201003111556.o2BFumCr006334@pork.ICSI.Berkeley.EDU> References: <201003111556.o2BFumCr006334@pork.ICSI.Berkeley.EDU> Message-ID: <4B993E10.6070707@vanderbilt.edu> What is the correct way to tell bro to ignore any traffic between two machines? If I have World, Machine1, and Machine2, then I want to monitor: World <-> Machine1 World <-> Machine2 but not Machine1 <-> Machine2. Thanks. Mat From hall.692 at osu.edu Thu Mar 11 11:44:12 2010 From: hall.692 at osu.edu (Seth Hall) Date: Thu, 11 Mar 2010 14:44:12 -0500 Subject: [Bro] Proper syntax for ignoring subnet to subnet traffic In-Reply-To: <4B993E10.6070707@vanderbilt.edu> References: <201003111556.o2BFumCr006334@pork.ICSI.Berkeley.EDU> <4B993E10.6070707@vanderbilt.edu> Message-ID: <4AE46E1A-9CCB-47F7-B03F-6E127DB9E554@osu.edu> On Mar 11, 2010, at 2:01 PM, Mathew Binkley wrote: > but not Machine1 <-> Machine2. Thanks. redef restrict_filters += { ["ignore_machine1_to_machine2"] = "not (host 1.2.3.4 and host 1.2.3.5)" }; Don't do that inside of an event handler or function definition. .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From mathew.binkley at vanderbilt.edu Thu Mar 11 12:10:12 2010 From: mathew.binkley at vanderbilt.edu (Mathew Binkley) Date: Thu, 11 Mar 2010 14:10:12 -0600 Subject: [Bro] Proper syntax for ignoring subnet to subnet traffic In-Reply-To: <4AE46E1A-9CCB-47F7-B03F-6E127DB9E554@osu.edu> References: <201003111556.o2BFumCr006334@pork.ICSI.Berkeley.EDU> <4B993E10.6070707@vanderbilt.edu> <4AE46E1A-9CCB-47F7-B03F-6E127DB9E554@osu.edu> Message-ID: <4B994E24.1050902@vanderbilt.edu> Thanks Seth. A slightly different question: how do I ignore traffic between hosts in a particular subnet? I want to ignore all chatter between machines in my cluster, and simply examine traffic between the cluster and the world. Mat From hall.692 at osu.edu Thu Mar 11 13:07:44 2010 From: hall.692 at osu.edu (Seth Hall) Date: Thu, 11 Mar 2010 16:07:44 -0500 Subject: [Bro] Proper syntax for ignoring subnet to subnet traffic In-Reply-To: <4B994E24.1050902@vanderbilt.edu> References: <201003111556.o2BFumCr006334@pork.ICSI.Berkeley.EDU> <4B993E10.6070707@vanderbilt.edu> <4AE46E1A-9CCB-47F7-B03F-6E127DB9E554@osu.edu> <4B994E24.1050902@vanderbilt.edu> Message-ID: On Mar 11, 2010, at 3:10 PM, Mathew Binkley wrote: > Thanks Seth. A slightly different question: how do I ignore traffic > between hosts in a particular subnet? I want to ignore all chatter > between machines in my cluster, and simply examine traffic between the > cluster and the world. I would do something similar to the earlier filter... redef restrict_filters += { ["ignore_internal"] = "not (src net 1.2.3.0/24 and dst net 1.2.3.0/24)" }; .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From estrada.veronica at gmail.com Fri Mar 12 08:06:55 2010 From: estrada.veronica at gmail.com (Veronica Estrada) Date: Sat, 13 Mar 2010 01:06:55 +0900 Subject: [Bro] linking a wrong_fragment event to a connection In-Reply-To: <201003111556.o2BFumCr006334@pork.ICSI.Berkeley.EDU> References: <201003111556.o2BFumCr006334@pork.ICSI.Berkeley.EDU> Message-ID: Thanks for your kindly answer about fragment. So maybe I should focus on bad_TCP_checksum / bad_UDP_checksum and bad_icmp_checksum instead of excessively_large_fragment, excessively_small_fragment, fragment_inconsistency, fragment_overlap,fragment_protocol_inconsistency etc..)? In that case, I will have the header with the information I need. BTW, in the case of BRO processed KDDCup 99 data, all the wrong fragments belong to SF labeled UDP/ICMP connections (for UDP connections wrong fragment count is 3 and the others 1). I thought SF is printed for completely flows but I read in an old message of this list (nov 2004) that when BRO encounters a flow mid-stream and that flow get shut down BRO uses SF label. You mentioned plans for adding Bro state tracking to solve it. What is the situation at the moment? The paper I am talking about defines the timeouts that could affect the probability to split a flow during its lifetime: active timeout is full length of TCP connection passive timeout refers to idle times in active flows where no packet is transfer (it is called also the maximum packet gap). Thank you in advance for any help you can provide. Veronica On Fri, Mar 12, 2010 at 12:56 AM, Vern Paxson wrote: > > How can I know which connection has generated that wrong fragment event? > > In general you can't. Most individual fragments do not possess transport > headers, so there is no well-defined connection associated with them. > *If* the fragment is part of a whole collection, then you can locate the > transport header at the beginning of the collection and use its ports - > but Bro may not have even seen this first part yet. So it only reports > the involved hosts. (It also could in principle report ports if the > problematic fragment happens to be the first *and* includes a full > transport > header [which it doesn't have to], but Bro doesn't go out of its way to > do the extra work in this case.) > > > By the way, I read about active and passive timeouts on connections > > ("Flow-based TCP Connection Analysis" by Limmer and Dressler). > > I don't understand how this topic is treated in BRO. > > You'll need to explain how those timeouts are defined in that paper for > others to be able to comment on how Bro's connection timeouts relate to > them. > > Vern > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20100313/9e38e54b/attachment.html From JAzoff at uamail.albany.edu Wed Mar 17 07:02:41 2010 From: JAzoff at uamail.albany.edu (Justin Azoff) Date: Wed, 17 Mar 2010 10:02:41 -0400 Subject: [Bro] Bro Memory Consumtion In-Reply-To: References: Message-ID: <20100317140241.GC14378@datacomm.albany.edu> On Mon, Feb 22, 2010 at 10:48:39AM -0500, Powell, Scott wrote: > Bro Gurus, > > I am having an issue with Bro and memory exhaustion. .. > I am running a large number of scripts from Seth Hall's script repository in > addition to the ones that are enabled by default. Below are the policies I'm > loading in local.bro: Hi all, following up on this.. 4 days ago I merged my bro policy with the latest updates from Seth, and since then the memory usage on my bro machine has flatlined(it used to look like a sawtooth wave from being restarted all the time). I'm not sure what the cause was, but my guess is something to do with the http file identification. The latest version of the script uses bro signatures instead of libmagic for file identification. I wonder if the libmagic code has a memory leak in it somewhere? If you are still having memory problems it would be really interesting to see if updating fixes things for you as well. -- -- Justin Azoff -- Network Security & Performance Analyst From hall.692 at osu.edu Wed Mar 17 08:21:57 2010 From: hall.692 at osu.edu (Seth Hall) Date: Wed, 17 Mar 2010 11:21:57 -0400 Subject: [Bro] Bro Memory Consumtion In-Reply-To: <20100317140241.GC14378@datacomm.albany.edu> References: <20100317140241.GC14378@datacomm.albany.edu> Message-ID: <0F9DA0C3-6393-490D-8EA9-BC9EE8FFAA9E@osu.edu> On Mar 17, 2010, at 10:02 AM, Justin Azoff wrote: > I wonder if the libmagic code has a memory leak in it > somewhere? Doh! It's certainly possible (I wrote the BiF for that). I'll write a test for it today. .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From powellsm at musc.edu Thu Mar 18 06:25:05 2010 From: powellsm at musc.edu (Powell, Scott) Date: Thu, 18 Mar 2010 09:25:05 -0400 Subject: [Bro] Bro Memory Consumtion In-Reply-To: <0F9DA0C3-6393-490D-8EA9-BC9EE8FFAA9E@osu.edu> References: <20100317140241.GC14378@datacomm.albany.edu> <0F9DA0C3-6393-490D-8EA9-BC9EE8FFAA9E@osu.edu> Message-ID: I synced my scripts up with the latest and greatest from Seth's repository but am still seeing Bro consume all 16gb of memory after only an hour or two. When time permits I will try to debug further to see if I can narrow it down to a particular script/policy. -----Original Message----- From: Seth Hall [mailto:hall.692 at osu.edu] Sent: Wednesday, March 17, 2010 11:22 AM To: Justin Azoff Cc: Powell, Scott; bro at ICSI.Berkeley.EDU Subject: Re: [Bro] Bro Memory Consumtion On Mar 17, 2010, at 10:02 AM, Justin Azoff wrote: > I wonder if the libmagic code has a memory leak in it > somewhere? Doh! It's certainly possible (I wrote the BiF for that). I'll write a test for it today. .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From JAzoff at uamail.albany.edu Thu Mar 18 06:27:10 2010 From: JAzoff at uamail.albany.edu (Justin Azoff) Date: Thu, 18 Mar 2010 09:27:10 -0400 Subject: [Bro] Bro Memory Consumtion In-Reply-To: References: <20100317140241.GC14378@datacomm.albany.edu> <0F9DA0C3-6393-490D-8EA9-BC9EE8FFAA9E@osu.edu> Message-ID: <20100318132710.GI5634@datacomm.albany.edu> On Thu, Mar 18, 2010 at 09:25:05AM -0400, Powell, Scott wrote: > I synced my scripts up with the latest and greatest from Seth's repository > but am still seeing Bro consume all 16gb of memory after only an hour or two. > When time permits I will try to debug further to see if I can narrow it down > to a particular script/policy. I forgot to mention, the name of the policy for the file detection changed.. Are you still loading http-identified-files or are you loading http-ext-identified-files? -- -- Justin Azoff -- Network Security & Performance Analyst From powellsm at musc.edu Thu Mar 18 06:30:52 2010 From: powellsm at musc.edu (Powell, Scott) Date: Thu, 18 Mar 2010 09:30:52 -0400 Subject: [Bro] Bro Memory Consumtion In-Reply-To: <20100318132710.GI5634@datacomm.albany.edu> References: <20100317140241.GC14378@datacomm.albany.edu> <0F9DA0C3-6393-490D-8EA9-BC9EE8FFAA9E@osu.edu> <20100318132710.GI5634@datacomm.albany.edu> Message-ID: I am loading the new one (http-ext-identified-files). I completely removed the old script as well as its @load statement. Here are the scripts of Seth's that I'm currently running: @load dns-passive-replication @load http-ext-identified-files @load http-hash @load known-hosts @load known-services @load logging.ftp-ext @load logging.http-ext @load logging.smtp-ext @load logging.ssh-ext @load smtp-ext-count-rejects @load software-ext @load ssh-ext @load ssl-ext -Scott -----Original Message----- From: Justin Azoff [mailto:JAzoff at uamail.albany.edu] Sent: Thursday, March 18, 2010 9:27 AM To: Powell, Scott Cc: Seth Hall; bro at ICSI.Berkeley.EDU Subject: Re: [Bro] Bro Memory Consumtion On Thu, Mar 18, 2010 at 09:25:05AM -0400, Powell, Scott wrote: > I synced my scripts up with the latest and greatest from Seth's repository > but am still seeing Bro consume all 16gb of memory after only an hour or two. > When time permits I will try to debug further to see if I can narrow it down > to a particular script/policy. I forgot to mention, the name of the policy for the file detection changed.. Are you still loading http-identified-files or are you loading http-ext-identified-files? -- -- Justin Azoff -- Network Security & Performance Analyst From hall.692 at osu.edu Thu Mar 18 06:37:03 2010 From: hall.692 at osu.edu (Seth Hall) Date: Thu, 18 Mar 2010 09:37:03 -0400 Subject: [Bro] Bro Memory Consumtion In-Reply-To: References: <20100317140241.GC14378@datacomm.albany.edu> <0F9DA0C3-6393-490D-8EA9-BC9EE8FFAA9E@osu.edu> Message-ID: <56428329-7544-4634-9FFA-35D6FA2817DB@osu.edu> On Mar 18, 2010, at 9:25 AM, Powell, Scott wrote: > I synced my scripts up with the latest and greatest from Seth's > repository but am still seeing Bro consume all 16gb of memory after > only an hour or two. When time permits I will try to debug further > to see if I can narrow it down to a particular script/policy. If it's using that much memory that quickly, my guess would be that there is a state table growing out of control. Load the "profiling" script, it will print out globals sizes every 20 minutes or so in a file named prof.log, then and you'll be able to see what variable(s) is/are so huge. If you are able to find out what variable is causing the memory consumption issue, please reply and let us know. It may be an issue that needs to be resolved or at least addressed in some way. .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From hall.692 at osu.edu Thu Mar 18 06:46:39 2010 From: hall.692 at osu.edu (Seth Hall) Date: Thu, 18 Mar 2010 09:46:39 -0400 Subject: [Bro] Bro Memory Consumtion In-Reply-To: References: <20100317140241.GC14378@datacomm.albany.edu> <0F9DA0C3-6393-490D-8EA9-BC9EE8FFAA9E@osu.edu> <20100318132710.GI5634@datacomm.albany.edu> Message-ID: <5CA859D9-1276-452D-9B09-B611C9F5EDF8@osu.edu> Instead of loading each of the "logging." scripts, you could just load enable-ext-logging at the top. Oh! I think I just noticed your problem (and it's my fault!). Remove dns-passive-replication.bro from your list of scripts and I think your memory problems will go away. The two dns scripts need work still. I may merge the two together at some point, but they don't clean up after themselves very well yet and they *do* cause bad memory consumption problems. Sorry about that! I really need to get all of the documentation written for my scripts. :) .Seth On Mar 18, 2010, at 9:30 AM, Powell, Scott wrote: > I am loading the new one (http-ext-identified-files). I completely > removed the old script as well as its @load statement. > > Here are the scripts of Seth's that I'm currently running: > > @load dns-passive-replication > @load http-ext-identified-files > @load http-hash > @load known-hosts > @load known-services > @load logging.ftp-ext > @load logging.http-ext > @load logging.smtp-ext > @load logging.ssh-ext > @load smtp-ext-count-rejects > @load software-ext > @load ssh-ext > @load ssl-ext > > -Scott > > -----Original Message----- > From: Justin Azoff [mailto:JAzoff at uamail.albany.edu] > Sent: Thursday, March 18, 2010 9:27 AM > To: Powell, Scott > Cc: Seth Hall; bro at ICSI.Berkeley.EDU > Subject: Re: [Bro] Bro Memory Consumtion > > On Thu, Mar 18, 2010 at 09:25:05AM -0400, Powell, Scott wrote: >> I synced my scripts up with the latest and greatest from Seth's >> repository >> but am still seeing Bro consume all 16gb of memory after only an >> hour or two. >> When time permits I will try to debug further to see if I can >> narrow it down >> to a particular script/policy. > > I forgot to mention, the name of the policy for the file detection > changed.. > Are you still loading http-identified-files or are you loading > http-ext-identified-files? > > -- > -- Justin Azoff > -- Network Security & Performance Analyst --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From hall.692 at osu.edu Thu Mar 18 06:51:19 2010 From: hall.692 at osu.edu (Seth Hall) Date: Thu, 18 Mar 2010 09:51:19 -0400 Subject: [Bro] Bro Memory Consumtion In-Reply-To: References: <20100317140241.GC14378@datacomm.albany.edu> <0F9DA0C3-6393-490D-8EA9-BC9EE8FFAA9E@osu.edu> Message-ID: On Mar 18, 2010, at 9:25 AM, Powell, Scott wrote: > I synced my scripts up with the latest and greatest from Seth's > repository but am still seeing Bro consume all 16gb of memory after > only an hour or two. When time permits I will try to debug further > to see if I can narrow it down to a particular script/policy. I just moved both of the dns scripts into the testing/ directory to clear up any confusion about their stability. :) When I get time and make them better with memory I'll move them back to the main directory. .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From powellsm at musc.edu Thu Mar 18 10:25:50 2010 From: powellsm at musc.edu (Powell, Scott) Date: Thu, 18 Mar 2010 13:25:50 -0400 Subject: [Bro] Bro Memory Consumtion In-Reply-To: References: <20100317140241.GC14378@datacomm.albany.edu> <0F9DA0C3-6393-490D-8EA9-BC9EE8FFAA9E@osu.edu> Message-ID: Seth, Thanks. I'm now running without the DNS scripts and have profiling enabled. I will see how it goes. Right now Bro is using about 4.5GB between the manager, proxy and my three workers (all running on the same system w/click splitting up the tap). I was restarting each day at 1am but I have commented out the cron. I'll check it in the morning and see if things are cleaning up after themselves. Thanks, Scott -----Original Message----- From: Seth Hall [mailto:hall.692 at osu.edu] Sent: Thursday, March 18, 2010 9:51 AM To: Powell, Scott Cc: Justin Azoff; bro at ICSI.Berkeley.EDU Subject: Re: [Bro] Bro Memory Consumtion On Mar 18, 2010, at 9:25 AM, Powell, Scott wrote: > I synced my scripts up with the latest and greatest from Seth's > repository but am still seeing Bro consume all 16gb of memory after > only an hour or two. When time permits I will try to debug further > to see if I can narrow it down to a particular script/policy. I just moved both of the dns scripts into the testing/ directory to clear up any confusion about their stability. :) When I get time and make them better with memory I'll move them back to the main directory. .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From vern at icir.org Thu Mar 18 10:46:33 2010 From: vern at icir.org (Vern Paxson) Date: Thu, 18 Mar 2010 10:46:33 -0700 Subject: [Bro] linking a wrong_fragment event to a connection In-Reply-To: (Sat, 13 Mar 2010 01:06:55 +0900). Message-ID: <201003181746.o2IHkXKr009660@pork.ICSI.Berkeley.EDU> > Thanks for your kindly answer about fragment. So maybe I should focus on > bad_TCP_checksum / bad_UDP_checksum and bad_icmp_checksum instead > of excessively_large_fragment, excessively_small_fragment, > fragment_inconsistency, > fragment_overlap,fragment_protocol_inconsistency > etc..)? In that case, I will have the header with the information I need. Yes, those all have some sort of flow information associated with them. (Though I would skip bad_icmp_checksum, as for ICMPs the flow is a somewhat vague notion.) > BTW, in the case of BRO processed KDDCup 99 data Side note: I hope you're only looking at that data get a handle on using Bro or such. Using it for research is a well-known pitfall; see http://www.icir.org/vern/talks/vp-IDS-Pitfalls-DIMVA07.pdf . > when BRO encounters a flow mid-stream and that flow get shut down BRO uses > SF label. You mentioned plans for adding Bro state tracking to solve it. > What is the situation at the moment? It's available, see the discussion "If you set the new control variable record_state_history to T" in the CHANGES file. > The paper I am talking about defines the timeouts that could affect the > probability to split a flow during its lifetime: > active timeout is full length of TCP connection passive timeout refers to > idle times in active flows where no packet is transfer (it is called also > the maximum packet gap). Bro does the latter, since it needs to make the decision in real-time. Vern From powellsm at musc.edu Fri Mar 19 10:50:29 2010 From: powellsm at musc.edu (Powell, Scott) Date: Fri, 19 Mar 2010 13:50:29 -0400 Subject: [Bro] Bro Memory Consumtion In-Reply-To: References: <20100317140241.GC14378@datacomm.albany.edu> <0F9DA0C3-6393-490D-8EA9-BC9EE8FFAA9E@osu.edu> Message-ID: Seth and all, My memory consumption is better but still growing and not shrinking. I've been examining the globals in the prof.log files for each of the various components (workers, manager, etc.) but am not sure what is causing so much memory to be allocated. Below is an example from one of my workers. There is ~3.6GB of memory allocated, total, but the globals are only 214MB. This is replicated across my three workers... plus the memory being used by the manager and proxy... so grand total I'm now up to ~12GB of allocated memory and it continues to grow. Mar 19 13:42:20 ------------------------ Mar 19 13:42:20 Memory: total=3821576K total_adj=3765668K malloced: 3814486K Mar 19 13:42:20 Run-time: user+sys=55905.4 user=53574.3 sys=2331.1 real=99872.4 Mar 19 13:42:20 Conns: total=12370755 current=4998/859 ext=0 mem=3372528K avg=3926.1 table=3430K connvals=2328K Mar 19 13:42:20 ConnCompressor: pending=36 pending_in_mem=582 full_conns=-4895 pending+real=4175 mem=48K avg=1368.7/84.7 Mar 19 13:42:20 Conns: tcp=0/0 udp=844/1984 icmp=15/50 Mar 19 13:42:20 TCP-States: Inact. Syn. SA Part. Est. Fin. Rst. Mar 19 13:42:20 TCP-States:Inact. 76 2 4 Mar 19 13:42:20 TCP-States:Syn. Mar 19 13:42:20 TCP-States:SA Mar 19 13:42:20 TCP-States:Part. 12 755 1 26 Mar 19 13:42:20 TCP-States:Est. 2412 98 7 Mar 19 13:42:20 TCP-States:Fin. 8 5 62 416 3 Mar 19 13:42:20 TCP-States:Rst. 95 12 90 54 1 Mar 19 13:42:20 Connections expired due to inactivity: 2012770 Mar 19 13:42:20 Total reassembler data: 236K Mar 19 13:42:20 RuleMatcher: matchers=2 dfa_states=599 ncomputed=9765 mem=1309K avg_nfa_states=19 Mar 19 13:42:20 Timers: current=12852 max=19240 mem=1004K lag=0.00s Mar 19 13:42:20 ConnectionDeleteTimer = 590 Mar 19 13:42:20 ConnectionInactivityTimer = 6874 Mar 19 13:42:20 DNSExpireTimer = 385 Mar 19 13:42:20 NetworkTimer = 1 Mar 19 13:42:20 NTPExpireTimer = 60 Mar 19 13:42:20 RotateTimer = 35 Mar 19 13:42:20 ScheduleTimer = 840 Mar 19 13:42:20 TableValTimer = 79 Mar 19 13:42:20 TCPConnectionAttemptTimer = 255 Mar 19 13:42:20 TCPConnectionExpireTimer = 3733 Mar 19 13:42:20 Global_sizes > 100k: 0K Mar 19 13:42:20 SSH::did_ssh_version = 24K (109/109 entries) Mar 19 13:42:20 Login::login_sessions = 122K (140/140 entries) Mar 19 13:42:20 SMTP::smtp_sessions = 973K (17/17 entries) Mar 19 13:42:20 KnownServices::established_conns = 191K (386/386 entries) Mar 19 13:42:20 ssl_cipher_desc = 30K (106/106 entries) Mar 19 13:42:20 dpd_analyzer_ports = 128K (35/700 entries) Mar 19 13:42:20 Scan::rops_idx = 39K (171/171 entries) Mar 19 13:42:20 notice_tags = 262K (690/690 entries) Mar 19 13:42:20 KnownHosts::known_hosts = 1861K (14160/14160 entries) Mar 19 13:42:20 Login::output_trouble = 399K Mar 19 13:42:20 DNS::distinct_PTR_requests = 481K (648/648 entries) Mar 19 13:42:20 Scan::distinct_ports = 5880K (5376/20084 entries) Mar 19 13:42:20 HTTP::http_sessions = 9018K (1697/1697 entries) Mar 19 13:42:20 ssl_connections = 2436K (905/905 entries) Mar 19 13:42:20 ftp_cmd_reply_code = 40K (273/273 entries) Mar 19 13:42:20 Weird::weird_ignore = 99K (94/188 entries) Mar 19 13:42:20 DNS::distinct_answered_PTR_requests = 45K (145/145 entries) Mar 19 13:42:20 SMTP::reject_counter = 5115K (9475/9475 entries) Mar 19 13:42:20 Scan::distinct_backscatter_peers = 269K (126/724 entries) Mar 19 13:42:20 DetectProtocolHTTP::conns = 438K (470/940 entries) Mar 19 13:42:20 HTTP::sql_injection_regex = 603K Mar 19 13:42:20 Scan::accounts_tried = 94K (96/222 entries) Mar 19 13:42:20 Portmapper::rpc_programs = 35K (129/129 entries) Mar 19 13:42:20 HTTP::known_user_agents = 10475K (8027/29020 entries) Mar 19 13:42:20 Scan::possible_scan_sources = 14K (106/106 entries) Mar 19 13:42:20 IRC::active_channels = 334K (47/47 entries) Mar 19 13:42:20 ssl_sessionIDs = 117981K (27276/27276 entries) Mar 19 13:42:20 FTP::hot_files = 112K Mar 19 13:42:20 Scan::pre_distinct_peers = 31560K (35230/72640 entries) Mar 19 13:42:20 HTTP::sensitive_URIs = 519K Mar 19 13:42:20 DetectProtocolHTTP::protocols = 278K (7/7 entries) Mar 19 13:42:20 Scan::distinct_low_ports = 89K (98/196 entries) Mar 19 13:42:20 IRC::active_users = 525K (96/96 entries) Mar 19 13:42:20 Scan::scan_triples = 7386K (106/17547 entries) Mar 19 13:42:20 Software::host_software = 9502K (5079/10272 entries) Mar 19 13:42:20 DNS::dns_sessions = 1011K (629/629 entries) Mar 19 13:42:20 Scan::distinct_peers = 4584K (571/30458 entries) Mar 19 13:42:20 HTTP::suspicious_http_posts = 733K Mar 19 13:42:20 KnownServices::known_services = 42K (261/261 entries) Mar 19 13:42:20 Login::input_trouble = 108K Mar 19 13:42:20 Weird::weird_action = 39K (170/170 entries) Mar 19 13:42:20 HTTP::conn_info = 3007K (759/759 entries) Mar 19 13:42:20 Global_sizes total: 219225K Mar 19 13:42:20 Total number of table entries: 115411/243104 Mar 19 13:42:35 ------------------------ PID 5562 (manager): 74112K 72.375M PID 5932 (manager): 201492K 196.77M PID 5950 (manager): 96240K 93.9844M PID 5962 (proxy-1): 74112K 72.375M PID 5974 (proxy-1): 144396K 141.012M PID 5975 (proxy-1): 97824K 95.5312M PID 6002 (worker-1): 74112K 72.375M PID 6038 (worker-1): 3608784K 3524.2M PID 6042 (worker-1): 94884K 92.6602M PID 5999 (worker-2): 74112K 72.375M PID 6036 (worker-2): 3966504K 3873.54M PID 6040 (worker-2): 94888K 92.6641M PID 6001 (worker-3): 74112K 72.375M PID 6037 (worker-3): 3930396K 3838.28M PID 6041 (worker-3): 93916K 91.7148M Total: 12.1116G Any ideas? Thanks, Scott -----Original Message----- From: bro-bounces at ICSI.Berkeley.EDU [mailto:bro-bounces at ICSI.Berkeley.EDU] On Behalf Of Powell, Scott Sent: Thursday, March 18, 2010 1:26 PM To: Seth Hall Cc: Justin Azoff; bro at ICSI.Berkeley.EDU Subject: Re: [Bro] Bro Memory Consumtion Seth, Thanks. I'm now running without the DNS scripts and have profiling enabled. I will see how it goes. Right now Bro is using about 4.5GB between the manager, proxy and my three workers (all running on the same system w/click splitting up the tap). I was restarting each day at 1am but I have commented out the cron. I'll check it in the morning and see if things are cleaning up after themselves. Thanks, Scott -----Original Message----- From: Seth Hall [mailto:hall.692 at osu.edu] Sent: Thursday, March 18, 2010 9:51 AM To: Powell, Scott Cc: Justin Azoff; bro at ICSI.Berkeley.EDU Subject: Re: [Bro] Bro Memory Consumtion On Mar 18, 2010, at 9:25 AM, Powell, Scott wrote: > I synced my scripts up with the latest and greatest from Seth's > repository but am still seeing Bro consume all 16gb of memory after > only an hour or two. When time permits I will try to debug further > to see if I can narrow it down to a particular script/policy. I just moved both of the dns scripts into the testing/ directory to clear up any confusion about their stability. :) When I get time and make them better with memory I'll move them back to the main directory. .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jones at tacc.utexas.edu Fri Mar 19 12:58:26 2010 From: jones at tacc.utexas.edu (William Jones) Date: Fri, 19 Mar 2010 14:58:26 -0500 Subject: [Bro] Bro Memory Consumtion In-Reply-To: References: <20100317140241.GC14378@datacomm.albany.edu> <0F9DA0C3-6393-490D-8EA9-BC9EE8FFAA9E@osu.edu> Message-ID: I think your memory usage is not too bad. Here is mine based on the output of top: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 19954 root 16 0 575m 512m 3264 R 36.1 6.4 2786:58 bro 19959 root 16 0 608m 546m 3208 R 26.3 6.8 2836:23 bro 19957 root 16 0 1876m 213m 3044 R 25.3 2.7 798:22.93 bro 19960 root 15 0 2269m 1.3g 2972 S 13.8 16.2 583:47.36 bro 19955 root 15 0 2242m 1.5g 2940 R 13.5 19.7 598:52.38 bro 19956 root 15 0 4381m 1.4g 3116 R 13.5 17.7 3207:07 bro 19958 root 15 0 2240m 94m 2892 R 5.9 1.2 1483:13 bro 20261 root 21 5 81400 3184 464 R 5.3 0.0 47:04.14 bro 20254 root 20 5 81400 3832 464 R 3.9 0.0 32:29.51 bro 20257 root 21 5 81400 2996 464 R 3.6 0.0 30:25.10 bro 20258 root 20 5 82584 4156 464 R 3.0 0.1 32:05.68 bro 20259 root 20 5 81664 3784 464 R 3.0 0.0 32:17.27 bro 20260 root 20 5 82584 4240 464 R 3.0 0.1 33:45.70 bro 20256 root 20 5 82664 3292 464 R 2.0 0.0 30:29.67 bro 14161 root 15 0 12740 1096 820 R 0.3 0.0 0:00.21 top -----Original Message----- From: bro-bounces at ICSI.Berkeley.EDU [mailto:bro-bounces at ICSI.Berkeley.EDU] On Behalf Of Powell, Scott Sent: Friday, March 19, 2010 12:50 PM To: Powell, Scott; Seth Hall Cc: Justin Azoff; bro at ICSI.Berkeley.EDU Subject: Re: [Bro] Bro Memory Consumtion Seth and all, My memory consumption is better but still growing and not shrinking. I've been examining the globals in the prof.log files for each of the various components (workers, manager, etc.) but am not sure what is causing so much memory to be allocated. Below is an example from one of my workers. There is ~3.6GB of memory allocated, total, but the globals are only 214MB. This is replicated across my three workers... plus the memory being used by the manager and proxy... so grand total I'm now up to ~12GB of allocated memory and it continues to grow. Mar 19 13:42:20 ------------------------ Mar 19 13:42:20 Memory: total=3821576K total_adj=3765668K malloced: 3814486K Mar 19 13:42:20 Run-time: user+sys=55905.4 user=53574.3 sys=2331.1 real=99872.4 Mar 19 13:42:20 Conns: total=12370755 current=4998/859 ext=0 mem=3372528K avg=3926.1 table=3430K connvals=2328K Mar 19 13:42:20 ConnCompressor: pending=36 pending_in_mem=582 full_conns=-4895 pending+real=4175 mem=48K avg=1368.7/84.7 Mar 19 13:42:20 Conns: tcp=0/0 udp=844/1984 icmp=15/50 Mar 19 13:42:20 TCP-States: Inact. Syn. SA Part. Est. Fin. Rst. Mar 19 13:42:20 TCP-States:Inact. 76 2 4 Mar 19 13:42:20 TCP-States:Syn. Mar 19 13:42:20 TCP-States:SA Mar 19 13:42:20 TCP-States:Part. 12 755 1 26 Mar 19 13:42:20 TCP-States:Est. 2412 98 7 Mar 19 13:42:20 TCP-States:Fin. 8 5 62 416 3 Mar 19 13:42:20 TCP-States:Rst. 95 12 90 54 1 Mar 19 13:42:20 Connections expired due to inactivity: 2012770 Mar 19 13:42:20 Total reassembler data: 236K Mar 19 13:42:20 RuleMatcher: matchers=2 dfa_states=599 ncomputed=9765 mem=1309K avg_nfa_states=19 Mar 19 13:42:20 Timers: current=12852 max=19240 mem=1004K lag=0.00s Mar 19 13:42:20 ConnectionDeleteTimer = 590 Mar 19 13:42:20 ConnectionInactivityTimer = 6874 Mar 19 13:42:20 DNSExpireTimer = 385 Mar 19 13:42:20 NetworkTimer = 1 Mar 19 13:42:20 NTPExpireTimer = 60 Mar 19 13:42:20 RotateTimer = 35 Mar 19 13:42:20 ScheduleTimer = 840 Mar 19 13:42:20 TableValTimer = 79 Mar 19 13:42:20 TCPConnectionAttemptTimer = 255 Mar 19 13:42:20 TCPConnectionExpireTimer = 3733 Mar 19 13:42:20 Global_sizes > 100k: 0K Mar 19 13:42:20 SSH::did_ssh_version = 24K (109/109 entries) Mar 19 13:42:20 Login::login_sessions = 122K (140/140 entries) Mar 19 13:42:20 SMTP::smtp_sessions = 973K (17/17 entries) Mar 19 13:42:20 KnownServices::established_conns = 191K (386/386 entries) Mar 19 13:42:20 ssl_cipher_desc = 30K (106/106 entries) Mar 19 13:42:20 dpd_analyzer_ports = 128K (35/700 entries) Mar 19 13:42:20 Scan::rops_idx = 39K (171/171 entries) Mar 19 13:42:20 notice_tags = 262K (690/690 entries) Mar 19 13:42:20 KnownHosts::known_hosts = 1861K (14160/14160 entries) Mar 19 13:42:20 Login::output_trouble = 399K Mar 19 13:42:20 DNS::distinct_PTR_requests = 481K (648/648 entries) Mar 19 13:42:20 Scan::distinct_ports = 5880K (5376/20084 entries) Mar 19 13:42:20 HTTP::http_sessions = 9018K (1697/1697 entries) Mar 19 13:42:20 ssl_connections = 2436K (905/905 entries) Mar 19 13:42:20 ftp_cmd_reply_code = 40K (273/273 entries) Mar 19 13:42:20 Weird::weird_ignore = 99K (94/188 entries) Mar 19 13:42:20 DNS::distinct_answered_PTR_requests = 45K (145/145 entries) Mar 19 13:42:20 SMTP::reject_counter = 5115K (9475/9475 entries) Mar 19 13:42:20 Scan::distinct_backscatter_peers = 269K (126/724 entries) Mar 19 13:42:20 DetectProtocolHTTP::conns = 438K (470/940 entries) Mar 19 13:42:20 HTTP::sql_injection_regex = 603K Mar 19 13:42:20 Scan::accounts_tried = 94K (96/222 entries) Mar 19 13:42:20 Portmapper::rpc_programs = 35K (129/129 entries) Mar 19 13:42:20 HTTP::known_user_agents = 10475K (8027/29020 entries) Mar 19 13:42:20 Scan::possible_scan_sources = 14K (106/106 entries) Mar 19 13:42:20 IRC::active_channels = 334K (47/47 entries) Mar 19 13:42:20 ssl_sessionIDs = 117981K (27276/27276 entries) Mar 19 13:42:20 FTP::hot_files = 112K Mar 19 13:42:20 Scan::pre_distinct_peers = 31560K (35230/72640 entries) Mar 19 13:42:20 HTTP::sensitive_URIs = 519K Mar 19 13:42:20 DetectProtocolHTTP::protocols = 278K (7/7 entries) Mar 19 13:42:20 Scan::distinct_low_ports = 89K (98/196 entries) Mar 19 13:42:20 IRC::active_users = 525K (96/96 entries) Mar 19 13:42:20 Scan::scan_triples = 7386K (106/17547 entries) Mar 19 13:42:20 Software::host_software = 9502K (5079/10272 entries) Mar 19 13:42:20 DNS::dns_sessions = 1011K (629/629 entries) Mar 19 13:42:20 Scan::distinct_peers = 4584K (571/30458 entries) Mar 19 13:42:20 HTTP::suspicious_http_posts = 733K Mar 19 13:42:20 KnownServices::known_services = 42K (261/261 entries) Mar 19 13:42:20 Login::input_trouble = 108K Mar 19 13:42:20 Weird::weird_action = 39K (170/170 entries) Mar 19 13:42:20 HTTP::conn_info = 3007K (759/759 entries) Mar 19 13:42:20 Global_sizes total: 219225K Mar 19 13:42:20 Total number of table entries: 115411/243104 Mar 19 13:42:35 ------------------------ PID 5562 (manager): 74112K 72.375M PID 5932 (manager): 201492K 196.77M PID 5950 (manager): 96240K 93.9844M PID 5962 (proxy-1): 74112K 72.375M PID 5974 (proxy-1): 144396K 141.012M PID 5975 (proxy-1): 97824K 95.5312M PID 6002 (worker-1): 74112K 72.375M PID 6038 (worker-1): 3608784K 3524.2M PID 6042 (worker-1): 94884K 92.6602M PID 5999 (worker-2): 74112K 72.375M PID 6036 (worker-2): 3966504K 3873.54M PID 6040 (worker-2): 94888K 92.6641M PID 6001 (worker-3): 74112K 72.375M PID 6037 (worker-3): 3930396K 3838.28M PID 6041 (worker-3): 93916K 91.7148M Total: 12.1116G Any ideas? Thanks, Scott -----Original Message----- From: bro-bounces at ICSI.Berkeley.EDU [mailto:bro-bounces at ICSI.Berkeley.EDU] On Behalf Of Powell, Scott Sent: Thursday, March 18, 2010 1:26 PM To: Seth Hall Cc: Justin Azoff; bro at ICSI.Berkeley.EDU Subject: Re: [Bro] Bro Memory Consumtion Seth, Thanks. I'm now running without the DNS scripts and have profiling enabled. I will see how it goes. Right now Bro is using about 4.5GB between the manager, proxy and my three workers (all running on the same system w/click splitting up the tap). I was restarting each day at 1am but I have commented out the cron. I'll check it in the morning and see if things are cleaning up after themselves. Thanks, Scott -----Original Message----- From: Seth Hall [mailto:hall.692 at osu.edu] Sent: Thursday, March 18, 2010 9:51 AM To: Powell, Scott Cc: Justin Azoff; bro at ICSI.Berkeley.EDU Subject: Re: [Bro] Bro Memory Consumtion On Mar 18, 2010, at 9:25 AM, Powell, Scott wrote: > I synced my scripts up with the latest and greatest from Seth's > repository but am still seeing Bro consume all 16gb of memory after > only an hour or two. When time permits I will try to debug further > to see if I can narrow it down to a particular script/policy. I just moved both of the dns scripts into the testing/ directory to clear up any confusion about their stability. :) When I get time and make them better with memory I'll move them back to the main directory. .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From hall.692 at osu.edu Tue Mar 23 18:39:20 2010 From: hall.692 at osu.edu (Seth Hall) Date: Tue, 23 Mar 2010 21:39:20 -0400 Subject: [Bro] Bro Memory Consumtion In-Reply-To: References: <20100317140241.GC14378@datacomm.albany.edu> <0F9DA0C3-6393-490D-8EA9-BC9EE8FFAA9E@osu.edu> Message-ID: <83BED055-5A90-4E4B-A43A-55C412BF49B1@osu.edu> On Mar 19, 2010, at 1:50 PM, Powell, Scott wrote: > My memory consumption is better but still growing and not shrinking. > I've been examining the globals in the prof.log files for each of > the various components (workers, manager, etc.) but am not sure what > is causing so much memory to be allocated. Below is an example from > one of my workers. There is ~3.6GB of memory allocated, total, but > the globals are only 214MB. Your issue has been nagging me the past few days because I couldn't explain why your memory use is so high. Today I finally realized what it could be. Did you provide the '--enable-brov6' flag when you built Bro? Even more worthwhile, could you provide the full configure line you used when you built Bro? (it's in the config.log file in the directory extracted from the tar.gz) .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From powellsm at musc.edu Wed Mar 24 06:38:18 2010 From: powellsm at musc.edu (Powell, Scott) Date: Wed, 24 Mar 2010 09:38:18 -0400 Subject: [Bro] Bro Memory Consumtion In-Reply-To: <83BED055-5A90-4E4B-A43A-55C412BF49B1@osu.edu> References: <20100317140241.GC14378@datacomm.albany.edu> <0F9DA0C3-6393-490D-8EA9-BC9EE8FFAA9E@osu.edu> <83BED055-5A90-4E4B-A43A-55C412BF49B1@osu.edu> Message-ID: Seth, Yes, I did include '--enable-brov6' because we are getting ready to rollout IPv6 in or perimeter and I was also seeing messages from Bro that it was not compiled with IPv6 support (via "broctl diag"). Here are all the parameters passed to configure: $ ./configure --prefix=/var/local/bro-1.5.1 --enable-cluster --enable-int64 --enable-brov6 --no-create --no-recursion -Scott -----Original Message----- From: Seth Hall [mailto:hall.692 at osu.edu] Sent: Tuesday, March 23, 2010 9:39 PM To: Powell, Scott Cc: Justin Azoff; bro at ICSI.Berkeley.EDU Subject: Re: [Bro] Bro Memory Consumtion On Mar 19, 2010, at 1:50 PM, Powell, Scott wrote: > My memory consumption is better but still growing and not shrinking. > I've been examining the globals in the prof.log files for each of > the various components (workers, manager, etc.) but am not sure what > is causing so much memory to be allocated. Below is an example from > one of my workers. There is ~3.6GB of memory allocated, total, but > the globals are only 214MB. Your issue has been nagging me the past few days because I couldn't explain why your memory use is so high. Today I finally realized what it could be. Did you provide the '--enable-brov6' flag when you built Bro? Even more worthwhile, could you provide the full configure line you used when you built Bro? (it's in the config.log file in the directory extracted from the tar.gz) .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From hall.692 at osu.edu Wed Mar 24 06:53:36 2010 From: hall.692 at osu.edu (Seth Hall) Date: Wed, 24 Mar 2010 09:53:36 -0400 Subject: [Bro] Bro Memory Consumtion In-Reply-To: References: <20100317140241.GC14378@datacomm.albany.edu> <0F9DA0C3-6393-490D-8EA9-BC9EE8FFAA9E@osu.edu> <83BED055-5A90-4E4B-A43A-55C412BF49B1@osu.edu> Message-ID: On Mar 24, 2010, at 9:38 AM, Powell, Scott wrote: > Yes, I did include '--enable-brov6' because we are getting ready to > rollout IPv6 in or perimeter and I was also seeing messages from Bro > that it was not compiled with IPv6 support (via "broctl diag"). Rebuild Bro without brov6 and int64 for now. Currently when you enable IPv6, all IP addresses consume 128-bits of memory (even IPv4 addresses!). You can see that this is what's happening by looking at the line in your prof.log that starts with "Conns:". It indicates that memory consumed just by connection state is over 3G (3372528K). There has been talk about changing things around so that IPv4 addresses still only take up 32-bits of memory even when IPv6 is enabled, but I don't know where those discussions ended and I don't know how difficult of a change that would be to make. Maybe Robin or Vern will comment on that? :) The IPv6 code has not been tested all that well either, so it's also possible that there are some memory leaks or other bugs lurking that could lead to high memory use. .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From robin at icir.org Wed Mar 24 08:56:46 2010 From: robin at icir.org (Robin Sommer) Date: Wed, 24 Mar 2010 08:56:46 -0700 Subject: [Bro] Bro Memory Consumtion In-Reply-To: References: <20100317140241.GC14378@datacomm.albany.edu> <0F9DA0C3-6393-490D-8EA9-BC9EE8FFAA9E@osu.edu> <83BED055-5A90-4E4B-A43A-55C412BF49B1@osu.edu> Message-ID: <20100324155646.GS41048@icir.org> On Wed, Mar 24, 2010 at 09:53 -0400, you wrote: > There has been talk about changing things around so that IPv4 > addresses still only take up 32-bits of memory even when IPv6 is > enabled, but I don't know where those discussions ended and I don't > know how difficult of a change that would be to make. Yeah, that's quite tricky. There's a ticket summarizing an earlier discussion: http://tracker.icir.org/bro/ticket/68 Robin -- Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From powellsm at musc.edu Thu Mar 25 12:45:27 2010 From: powellsm at musc.edu (Powell, Scott) Date: Thu, 25 Mar 2010 15:45:27 -0400 Subject: [Bro] Bro Memory Consumtion In-Reply-To: References: <20100317140241.GC14378@datacomm.albany.edu> <0F9DA0C3-6393-490D-8EA9-BC9EE8FFAA9E@osu.edu> <83BED055-5A90-4E4B-A43A-55C412BF49B1@osu.edu> Message-ID: I recompiled without IPv6 and int64 today and so far my memory footprint is considerably lower, as expected. I will keep an eye on it over the next few days (I have disabled my nightly restart cron) and see how it behaves. We have just brought IPv6 to our border router and will soon be testing it in the perimeter. Hopefully by the time we get anywhere close to wide spread usage Bro will have better support for it. Wishful thinking, huh? :) -----Original Message----- From: Seth Hall [mailto:hall.692 at osu.edu] Sent: Wednesday, March 24, 2010 9:54 AM To: Powell, Scott Cc: Justin Azoff; bro at ICSI.Berkeley.EDU Subject: Re: [Bro] Bro Memory Consumtion On Mar 24, 2010, at 9:38 AM, Powell, Scott wrote: > Yes, I did include '--enable-brov6' because we are getting ready to > rollout IPv6 in or perimeter and I was also seeing messages from Bro > that it was not compiled with IPv6 support (via "broctl diag"). Rebuild Bro without brov6 and int64 for now. Currently when you enable IPv6, all IP addresses consume 128-bits of memory (even IPv4 addresses!). You can see that this is what's happening by looking at the line in your prof.log that starts with "Conns:". It indicates that memory consumed just by connection state is over 3G (3372528K). There has been talk about changing things around so that IPv4 addresses still only take up 32-bits of memory even when IPv6 is enabled, but I don't know where those discussions ended and I don't know how difficult of a change that would be to make. Maybe Robin or Vern will comment on that? :) The IPv6 code has not been tested all that well either, so it's also possible that there are some memory leaks or other bugs lurking that could lead to high memory use. .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From powellsm at musc.edu Mon Mar 29 05:16:18 2010 From: powellsm at musc.edu (Powell, Scott) Date: Mon, 29 Mar 2010 08:16:18 -0400 Subject: [Bro] Bro Memory Consumtion In-Reply-To: References: <20100317140241.GC14378@datacomm.albany.edu> <0F9DA0C3-6393-490D-8EA9-BC9EE8FFAA9E@osu.edu> <83BED055-5A90-4E4B-A43A-55C412BF49B1@osu.edu> Message-ID: Seth, I wanted to circle back around on this. This was definitely the issue as my memory usage has now flat lined. I have not restarted Bro in 4 days and my total memory usage is < 3GB for all workers, proxy and manager combined. Thanks for the help. -Scott -----Original Message----- From: bro-bounces at ICSI.Berkeley.EDU [mailto:bro-bounces at ICSI.Berkeley.EDU] On Behalf Of Powell, Scott Sent: Thursday, March 25, 2010 3:45 PM To: Seth Hall Cc: Justin Azoff; bro at ICSI.Berkeley.EDU Subject: Re: [Bro] Bro Memory Consumtion I recompiled without IPv6 and int64 today and so far my memory footprint is considerably lower, as expected. I will keep an eye on it over the next few days (I have disabled my nightly restart cron) and see how it behaves. We have just brought IPv6 to our border router and will soon be testing it in the perimeter. Hopefully by the time we get anywhere close to wide spread usage Bro will have better support for it. Wishful thinking, huh? :) -----Original Message----- From: Seth Hall [mailto:hall.692 at osu.edu] Sent: Wednesday, March 24, 2010 9:54 AM To: Powell, Scott Cc: Justin Azoff; bro at ICSI.Berkeley.EDU Subject: Re: [Bro] Bro Memory Consumtion On Mar 24, 2010, at 9:38 AM, Powell, Scott wrote: > Yes, I did include '--enable-brov6' because we are getting ready to > rollout IPv6 in or perimeter and I was also seeing messages from Bro > that it was not compiled with IPv6 support (via "broctl diag"). Rebuild Bro without brov6 and int64 for now. Currently when you enable IPv6, all IP addresses consume 128-bits of memory (even IPv4 addresses!). You can see that this is what's happening by looking at the line in your prof.log that starts with "Conns:". It indicates that memory consumed just by connection state is over 3G (3372528K). There has been talk about changing things around so that IPv4 addresses still only take up 32-bits of memory even when IPv6 is enabled, but I don't know where those discussions ended and I don't know how difficult of a change that would be to make. Maybe Robin or Vern will comment on that? :) The IPv6 code has not been tested all that well either, so it's also possible that there are some memory leaks or other bugs lurking that could lead to high memory use. .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From hall.692 at osu.edu Mon Mar 29 06:47:07 2010 From: hall.692 at osu.edu (Seth Hall) Date: Mon, 29 Mar 2010 09:47:07 -0400 Subject: [Bro] Bro Memory Consumtion In-Reply-To: References: <20100317140241.GC14378@datacomm.albany.edu> <0F9DA0C3-6393-490D-8EA9-BC9EE8FFAA9E@osu.edu> <83BED055-5A90-4E4B-A43A-55C412BF49B1@osu.edu> Message-ID: <0889F77F-8A91-4781-A40E-E041339E4521@osu.edu> On Mar 29, 2010, at 8:16 AM, Powell, Scott wrote: > This was definitely the issue as my memory usage has now flat lined. > I have not restarted Bro in 4 days and my total memory usage is < > 3GB for all workers, proxy and manager combined. Awesome! > Thanks for the help. No problem, I'm glad that helped. I'm taking a look at some of the IPv6 code now to see if there is anything I can do to help reduce memory usage because I'd also like to be able to run Bro with IPv6 enabled. .Seth From robin at icir.org Mon Mar 29 10:53:45 2010 From: robin at icir.org (Robin Sommer) Date: Mon, 29 Mar 2010 10:53:45 -0700 Subject: [Bro] Bro Memory Consumtion In-Reply-To: <0889F77F-8A91-4781-A40E-E041339E4521@osu.edu> References: <83BED055-5A90-4E4B-A43A-55C412BF49B1@osu.edu> <0889F77F-8A91-4781-A40E-E041339E4521@osu.edu> Message-ID: <20100329175345.GF93054@icir.org> On Mon, Mar 29, 2010 at 09:47 -0400, Seth Hall wrote: > No problem, I'm glad that helped. I'm taking a look at some of the > IPv6 code now to see if there is anything I can do to help reduce > memory usage because I'd also like to be able to run Bro with IPv6 > enabled. Another question would be if there might be any memory leaks in the v6 code. I don't think anybody has ever profiled that part of the code for leaks ... Robin -- Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From tarupp at fnal.gov Tue Mar 30 09:55:41 2010 From: tarupp at fnal.gov (Tim Rupp) Date: Tue, 30 Mar 2010 11:55:41 -0500 Subject: [Bro] ignoring all weird? Message-ID: <4BB22D0D.6030108@fnal.gov> Is there a convenient way that I can suppress all weird messages that would otherwise bubble up to the weird log? I've done this redef notice_action_filters += { [[Weird::WeirdActivity, Weird::ContentGap, Weird::RetransmissionInconsistency, Weird::AckAboveHole]] = ignore_notice }; But I still get some weird messages that I need to suppress like this redef Weird::weird_action: table[string] of Weird::WeirdAction += { [["above_hole_data_without_any_acks", "bad_TCP_checksum", "unmatched_HTTP_reply", "connection_originator_SYN_ack", "window_recision", "unescaped_special_URI_char", "bad_UDP_checksum", "data_before_established", "inflate_failed", "line_terminated_with_single_CR" ]] = Weird::WEIRD_IGNORE }; Ideas? Thanks, -Tim From JAzoff at uamail.albany.edu Tue Mar 30 11:02:00 2010 From: JAzoff at uamail.albany.edu (Justin Azoff) Date: Tue, 30 Mar 2010 14:02:00 -0400 Subject: [Bro] ignoring all weird? In-Reply-To: <4BB22D0D.6030108@fnal.gov> References: <4BB22D0D.6030108@fnal.gov> Message-ID: <20100330180159.GD3886@datacomm.albany.edu> On Tue, Mar 30, 2010 at 11:55:41AM -0500, Tim Rupp wrote: > Is there a convenient way that I can suppress all weird messages that > would otherwise bubble up to the weird log? try something like this: redef suppress_notice_actions += { Weird::ContentGap, Weird::AckAboveHole, Weird::WeirdActivity, }; -- -- Justin Azoff -- Network Security & Performance Analyst From raoofeh_h at yahoo.com Tue Mar 30 21:42:35 2010 From: raoofeh_h at yahoo.com (raoofeh_h at yahoo.com) Date: Tue, 30 Mar 2010 21:42:35 -0700 (PDT) Subject: [Bro] (no subject) Message-ID: <184480.75172.qm@web65509.mail.ac4.yahoo.com> Hello all, I am going to install bro in my system but i get the following error when making: FlowSrc.cc: In constructor `FlowSocketSrc::FlowSocketSrc(const char*)': FlowSrc.cc:142: error: `errno' was not declared in this scope FlowSrc.cc:150: error: `errno' was not declared in this scope FlowSrc.cc: In member function `virtual int FlowFileSrc::ExtractNextPDU()': FlowSrc.cc:171: error: `errno' was not declared in this scope FlowSrc.cc:179: error: `errno' was not declared in this scope FlowSrc.cc:185: error: `errno' was not declared in this scope FlowSrc.cc: In constructor `FlowFileSrc::FlowFileSrc(const char*)': FlowSrc.cc:217: error: `errno' was not declared in this scope make[3]: *** [FlowSrc.o] Error 1 make[3]: Leaving directory `/root/Desktop/bro-1.4/src' make[2]: *** [all] Error 2 make[2]: Leaving directory `/root/Desktop/bro-1.4/src' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/root/Desktop/bro-1.4' make: *** [all] Error 2 Would you please help me if you know what is the problem? Regards, Raoufeh -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20100330/e6d846a6/attachment.html From tarupp at fnal.gov Wed Mar 31 06:53:03 2010 From: tarupp at fnal.gov (Tim Rupp) Date: Wed, 31 Mar 2010 08:53:03 -0500 Subject: [Bro] ignoring all weird? In-Reply-To: <20100330180159.GD3886@datacomm.albany.edu> References: <4BB22D0D.6030108@fnal.gov> <20100330180159.GD3886@datacomm.albany.edu> Message-ID: <4BB353BF.8050902@fnal.gov> On 03/30/2010 01:02 PM, Justin Azoff wrote: > On Tue, Mar 30, 2010 at 11:55:41AM -0500, Tim Rupp wrote: >> Is there a convenient way that I can suppress all weird messages that >> would otherwise bubble up to the weird log? > > try something like this: > > redef suppress_notice_actions += { > Weird::ContentGap, > Weird::AckAboveHole, > Weird::WeirdActivity, > }; > I was hoping it'd be that easy, but now the weird messages (content gap, ack above a hole, etc) are going to stdout as well as the weird log. Ideas? Thanks, -Tim From vijay.m.khadse1979 at gmail.com Wed Mar 31 07:50:14 2010 From: vijay.m.khadse1979 at gmail.com (vijay khadse) Date: Wed, 31 Mar 2010 20:20:14 +0530 Subject: [Bro] DDOS ( SYN flodding attack) Message-ID: Can we use BRO to detect DDOS ( SYN Flodding attack) at a router. If yes how? I had a internet trace obtained from CAIDA ISP-A . I have to detect SYN flodding attacks in that trace. It is a PCAP file of 2GB. Please help me .I have tried it with snort but was not succesful. Please Help me. Thank you . Regards, Vijay M Khadse -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20100331/b0c6f92d/attachment.html From robin at icir.org Wed Mar 31 09:35:08 2010 From: robin at icir.org (Robin Sommer) Date: Wed, 31 Mar 2010 09:35:08 -0700 Subject: [Bro] (no subject) In-Reply-To: <184480.75172.qm@web65509.mail.ac4.yahoo.com> References: <184480.75172.qm@web65509.mail.ac4.yahoo.com> Message-ID: <20100331163508.GA39133@icir.org> On Tue, Mar 30, 2010 at 21:42 -0700, raoofeh_h at yahoo.com wrote: > FlowSrc.cc:142: error: `errno' was not declared in this scope Is that with Bro 1.5? On what platform are you? As a quick hack, you could try adding "#include " at the beginning of that file. Robin -- Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From vern at icir.org Wed Mar 31 09:35:13 2010 From: vern at icir.org (Vern Paxson) Date: Wed, 31 Mar 2010 09:35:13 -0700 Subject: [Bro] DDOS ( SYN flodding attack) In-Reply-To: (Wed, 31 Mar 2010 20:20:14 +0530). Message-ID: <201003311635.o2VGZDMo007228@pork.ICSI.Berkeley.EDU> > Can we use BRO to detect DDOS ( SYN Flodding attack) at a router. There's a policy script for doing this, synflood.bro - give it a try. Vern From robin at icir.org Wed Mar 31 09:47:04 2010 From: robin at icir.org (Robin Sommer) Date: Wed, 31 Mar 2010 09:47:04 -0700 Subject: [Bro] ignoring all weird? In-Reply-To: <4BB353BF.8050902@fnal.gov> References: <4BB22D0D.6030108@fnal.gov> <20100330180159.GD3886@datacomm.albany.edu> <4BB353BF.8050902@fnal.gov> Message-ID: <20100331164704.GB39133@icir.org> On Wed, Mar 31, 2010 at 08:53 -0500, Tim Rupp wrote: > I was hoping it'd be that easy, but now the weird messages (content gap, > ack above a hole, etc) are going to stdout as well as the weird log. I don't think we currently have a way to generally suppress all weird messages, just individually per your earlier mail. It is however posssible to remove them from notice.log by mapping corresponding notice types to NOTICE_IGNORE. Justin's solution is almost doing that but it just suppresses the notice_action_event, not the actual reporting (that event is used mostly internally). Use either notice_policy or notice_action_filters instead. Robin -- Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Wed Mar 31 09:48:32 2010 From: robin at icir.org (Robin Sommer) Date: Wed, 31 Mar 2010 09:48:32 -0700 Subject: [Bro] DDOS ( SYN flodding attack) In-Reply-To: References: Message-ID: <20100331164832.GC39133@icir.org> On Wed, Mar 31, 2010 at 20:20 +0530, vijay khadse wrote: > Can we use BRO to detect DDOS ( SYN Flodding attack) at a router. Try running Bro with the script synflood.bro. You may need to tweak a few of the parameters at the beginning of the script. Robin -- Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From sridhar.basam at gmail.com Wed Mar 31 09:49:01 2010 From: sridhar.basam at gmail.com (sridhar basam) Date: Wed, 31 Mar 2010 12:49:01 -0400 Subject: [Bro] DDOS ( SYN flodding attack) In-Reply-To: References: Message-ID: You might want to take a look at the policy file synflood.bro to see if it does what you are looking for or use it as a starting point to write a custom policy. Sri On Wed, Mar 31, 2010 at 10:50 AM, vijay khadse wrote: > Can we use BRO to detect DDOS ( SYN Flodding attack) at a router. If yes > how? I had a internet trace obtained from CAIDA ISP-A . I have to detect > SYN flodding attacks in that trace. It is a PCAP file of 2GB. Please help me > .I have tried it with snort but was not succesful. > Please Help me. Thank you . > > Regards, > Vijay M Khadse > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Sridhar -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20100331/ac2e0fb7/attachment.html From robin at icir.org Wed Mar 31 09:52:29 2010 From: robin at icir.org (Robin Sommer) Date: Wed, 31 Mar 2010 09:52:29 -0700 Subject: [Bro] (no subject) In-Reply-To: <20100331163508.GA39133@icir.org> References: <184480.75172.qm@web65509.mail.ac4.yahoo.com> <20100331163508.GA39133@icir.org> Message-ID: <20100331165229.GA40883@icir.org> On Wed, Mar 31, 2010 at 09:35 -0700, I wrote: > Is that with Bro 1.5? I've missed the 1.4 in the output you sent. You could try 1.5, but my guess is you'd see the same problem. Robin -- Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org