From jlay at slave-tothe-box.net Sun Feb 1 07:03:57 2015 From: jlay at slave-tothe-box.net (James Lay) Date: Sun, 01 Feb 2015 08:03:57 -0700 Subject: [Bro] Revisiting log rotate only In-Reply-To: <54BFD126.8020007@illinois.edu> References: <1421505437.3223.16.camel@JamesiMac> <1421675855.3196.3.camel@JamesiMac> <54BEB4D1.5060400@illinois.edu> <9a5b19652863fc2f4068ca2fcf1e1d5b@localhost> <54BED406.2070709@illinois.edu> <54BF1C8F.8020104@illinois.edu> <1421838101.3220.16.camel@JamesiMac> <54BFD126.8020007@illinois.edu> Message-ID: <1422803037.3071.2.camel@JamesiMac> On Wed, 2015-01-21 at 10:17 -0600, Daniel Thayer wrote: > On 01/21/2015 05:01 AM, James Lay wrote: > > On Tue, 2015-01-20 at 21:27 -0600, Daniel Thayer wrote: > >> On 01/20/2015 04:52 PM, James Lay wrote: > >> > On 2015-01-20 03:17 PM, Daniel Thayer wrote: > >> >> On 01/20/2015 04:13 PM, James Lay wrote: > >> >>> On 2015-01-20 01:04 PM, Daniel Thayer wrote: > >> >>>> On 01/19/2015 07:57 AM, James Lay wrote: > >> >>>>> On Sat, 2015-01-17 at 07:37 -0700, James Lay wrote: > >> >>>>>> Hey all, > >> >>>>>> > >> >>>>>> I posted about this last August here: > >> >>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>>>http://mailman.icsi.berkeley.edu/pipermail/bro/2014-August/007329.html > >> >>>>>> > >> >>>>>> I also noticed someone have a disappearing log event which I have > >> >>>>>> seen > >> >>>>>> before as well here: > >> >>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>>>http://mailman.icsi.berkeley.edu/pipermail/bro/2015-January/007935.html > >> >>>>>> > >> >>>>>> I documented my process on installing bro on Ubuntu 14.04 using > >> >>>>>> just > >> >>>>>> log rotation below: > >> >>>>>> > >> >>>>>> sudo apt-get -y install cmake > >> >>>>>> sudo apt-get -y install python-dev > >> >>>>>> sudo apt-get -y install swig > >> >>>>>> cp /usr/local/bro/share/bro/site > >> >>>>>> cp /opt/bin/startbro <- command line bro with long --filter line > >> >>>>>> cp /opt/bin/startbro to /etc/rc.local > >> >>>>>> sudo ln -s /usr/local/bro/bin/bro /usr/local/bin/ > >> >>>>>> sudo ln -s /usr/local/bro/bin/bro-cut /usr/local/bin/ > >> >>>>>> sudo ln -s /usr/local/bro/bin/broctl /usr/local/bin/ > >> >>>>>> sudo ln -s /usr/local/bro/share/broctl/scripts/archive-log > >> >>>>>> /usr/local/bin/ > >> >>>>>> sudo ln -s /usr/local/bro/share/broctl/scripts/broctl-config.sh > >> >>>>>> /usr/local/bin/ > >> >>>>>> sudo ln -s > >> >>>>>> /usr/local/bro/share/broctl/scripts/create-link-for-log > >> >>>>>> /usr/local/bin/ > >> >>>>>> sudo ln -s /usr/local/bro/share/broctl/scripts/make-archive-name > >> >>>>>> /usr/local/bin/ > >> >>>>>> git clonehttps://github.com/jonschipp/mal-dnssearch.git > >> >>>>>> sudo make install > >> >>>>>> > >> >>>>>> specifics on log rotate only: > >> >>>>>> > >> >>>>>> add the below to local.bro > >> >>>>>> redef Log::default_rotation_interval = 86400 secs; > >> >>>>>> redef Log::default_rotation_postprocessor_cmd = "archive-log"; > >> >>>>>> edit the below in broctl.cfg > >> >>>>>> MailTo =jlay at slave-tothe-box.net > >> >>>>>> > >> >>>>>> LogRotationInterval = 86400 > >> >>>>>> sudo /usr/local/bro/bin/broctl install > >> >>>>>> > >> >>>>>> Besides the edits to broctl.cfg, file locations are the default. > >> >>>>>> The > >> >>>>>> above works well usually...it's after a reboot I have found > >> >>>>>> things go > >> >>>>>> bad. Usually logs get rotated at midnight and I get an email > >> >>>>>> with > >> >>>>>> statistics, just what I need. I rebooted the machine on the 13, > >> >>>>>> and > >> >>>>>> that's the last email or log rotation I got....this morning I see > >> >>>>>> current has files and my logstash instance has data so I believe > >> >>>>>> the > >> >>>>>> rotation got..."stuck". I'm kicking myself for not > >> >>>>>> heading/tailing > >> >>>>>> the files first, but after issuing a "sudo killall bro", those > >> >>>>>> file in > >> >>>>>> current vanished, no directory was created, and I received no > >> >>>>>> email, > >> >>>>>> that data is now gone (no big deal as this is at home). I > >> >>>>>> decided to > >> >>>>>> run broctl install again, then start and kill bro one more time. > >> >>>>>> At > >> >>>>>> that point, I got a new directory with log rotation and an email > >> >>>>>> with > >> >>>>>> minutes or so of stats. Please let me know if there's something > >> >>>>>> I can > >> >>>>>> do on my end to trouble shoot. Thank you. > >> >>>>>> > >> >>>>>> James > >> >>>>>> _______________________________________________ > >> >>>>>> Bro mailing list > >> >>>>>>bro at bro-ids.org > >> >>>>>>http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > >> >>>>> > >> >>>>> Confirming that this method is no longer working. Heading my > >> >>>>> connlog > >> >>>>> file I see: > >> >>>>> > >> >>>>> #open 2015-01-19-00-00-05 > >> >>>>> > >> >>>>> my /usr/local/bro/logs is completely missing Jan 18th. From my > >> >>>>> broctl.cfg: > >> >>>>> > >> >>>>> SpoolDir = /usr/local/bro/spool > >> >>>>> LogDir = /usr/local/bro/logs > >> >>>>> LogRotationInterval = 86400 > >> >>>>> > >> >>>>> From my /usr/local/bro/share/bro/site/local.bro: > >> >>>>> > >> >>>>> redef Log::default_rotation_interval = 86400 secs; > >> >>>>> redef Log::default_rotation_postprocessor_cmd = "archive-log"; > >> >>>>> > >> >>>>> Anything else I can do to debug this? Thank you. > >> >>>>> > >> >>>>> James > >> >>>> > >> >>>> Are you using broctl to start and stop Bro? What does > >> >>>> /opt/bin/startbro > >> >>>> do? > >> >>> > >> >>> Thanks for looking Daniel. I am starting this with the below: > >> >>> > >> >>> /usr/local/bro/bin/bro --no-checksums -i eth0 -i ppp0 --filter '( > >> >>> large > >> >>> filter line here)' local "Site::local_nets += { 192.168.1.0/24 }" > >> >>> > >> >>> I'm not using broctl. The only small portion that I am is for the > >> >>> log > >> >>> rotation as outlined in the email thread. After killing and > >> >>> starting > >> >>> bro yesterday, this morning at midnight logs got rotated and I got > >> >>> my > >> >>> report email. This appears to happen after a complete reboot of the > >> >>> device. It's very odd. Thanks again. > >> >>> > >> >>> James > >> >> > >> >> What command do you use to stop (or restart) Bro? > >> > > >> > The classic: sudo killall bro :) when I have to do it manually. Then > >> > start with the command line above. Thanks again. > >> > > >> > James > >> > >> OK, since you're not using broctl to start/stop bro, here's > >> what happens: > >> > >> When you stop bro, bro will rotate all log files (rename them with > >> a timestamp). Then, bro will spawn "archive-log" processes, one > >> per log file, to archive (i.e., copy or gzip to another directory) > >> each rotated log file. This can take some time, depending on the > >> log file size, and whether you're generating connection summary > >> reports or not. If the machine is rebooted while this is > >> happening, then one or more of the rotated logs might not get > >> archived (because the "archive-log" processes were killed before > >> they had a chance to finish). > >> > >> Next time you boot your machine and start bro, the rotated logs will > >> still be there (unless you have some other script that removes that > >> directory), but they will never get archived automatically. > >> And, because the rotated log filenames contain a date/timestamp, they > >> will not be overwritten by new logs. > >> > >> To avoid this issue when you want to reboot, I suggest stopping bro, > >> and then waiting for all the logs to finish being archived, then reboot. > > > > Thanks Daniel, > > > > So compressed the entire directory of log files is 7.5 megs....really > > small, so I don't think it's a question of getting stuck during > > compression (truth be told the box doing the bro-ing is sitting right > > next to the box I'm typing this email on...I can hear the drive whir > > away when I stop bro and it lasts maybe 30 seconds). Also, before > > reboot I manually stop bro...out of habit. My only thought is that > > *maybe* the path of /usr/local/bin/ where I've symlinked the additional > > scripts aren't seen when my startbro script is run from /etc/rc.local > > file? In any case I can reproduce the behavior on reboot, so if there's > > a way to debug this I'd love to give it a go. I'll research the path > > thing on my end (Ubuntu 14.0.4) and I'll try a) rebooting and starting > > bro manually and b) symlinking the script files to /usr/local/sbin/. > > I'll report my findings for anyone else out there, but I kinda think > > most people are just using broctl anyways :) Thanks again Daniel. > > > > James > > > One other thing to check is which directory you are starting Bro from, > because that's where Bro will create its log files (if you were > using broctl, this should be /usr/local/bro/spool/bro). > > If you ever notice that you are missing logs in the archive directory > (a subdirectory of /usr/local/bro/logs), then you'll want to check > the directory where you were running Bro to see if it contains any > unarchived logs (if you were using broctl to start/stop bro, then > you'd also need to check all subdirectories of > /usr/local/bro/spool/tmp). So I think I may have this resolved. Yesterday I noticed that two symlinks were bad: create-link-for-log make-archive-name I've symlinked these correctly and rebooted. I manually started bro instead of having it start in /etc/rc.local. My last test when I need to reboot again will be to have bro autostart. Thanks all. James -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150201/9821ed29/attachment.html From liburdi.joshua at gmail.com Sun Feb 1 12:34:31 2015 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Sun, 1 Feb 2015 12:34:31 -0800 Subject: [Bro] Memory leak output Message-ID: Hey everyone, I'm performing testing for memory leaks and am running into an interesting perftools error message: Thread finding failed with -1 errno=1 Could not find thread stacks. Will likely report false leak positives. Have memory regions w/o callers: might report false leaks Leak check net_run detected leaks of 1520 bytes in 10 objects The 1 largest leaks: Leak of 1520 bytes in 10 objects allocated from: If the preceding stack traces are not enough to find the leaks, try running THIS shell command: pprof bro "/tmp/bro.3885.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 --gv I can't tell what caused this error. Has anyone seen this before? This is my first time running a memory leak test and this is happening for all trace files, so I'm not sure if this is normal (guessing no). Josh From luismiguelferreirasilva at gmail.com Sun Feb 1 21:35:25 2015 From: luismiguelferreirasilva at gmail.com (Luis Miguel Silva) Date: Sun, 1 Feb 2015 22:35:25 -0700 Subject: [Bro] Best practice on how to customize an officially distributed script Message-ID: Dear all, I would like to change the known-hosts.bro script to log both the ip and macaddr for all known hosts in my network. What are the best practices for customizing scripts that ship with bro (e.g. distributed in the /usr/share/bro/* directory)? Am I supposed to just: - copy the script I want to customize to my share/bro/site/ - and change local.bro to load the script in share/bro/site/ instead of share/bro/policy/protocols/conn/known-hosts.bro? Thank you, Luis -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150201/e77178cd/attachment.html From jdonnelly at dyn.com Mon Feb 2 04:57:07 2015 From: jdonnelly at dyn.com (John Donnelly) Date: Mon, 2 Feb 2015 06:57:07 -0600 Subject: [Bro] Memory leak output In-Reply-To: References: Message-ID: I have built Bro with the perftools enabled and get similar messages ; the pprof cli notice I've seen doesn't work either. On Sun, Feb 1, 2015 at 2:34 PM, Josh Liburdi wrote: > Hey everyone, > > I'm performing testing for memory leaks and am running into an > interesting perftools error message: > > Thread finding failed with -1 errno=1 > Could not find thread stacks. Will likely report false leak positives. > Have memory regions w/o callers: might report false leaks > Leak check net_run detected leaks of 1520 bytes in 10 objects > The 1 largest leaks: > Leak of 1520 bytes in 10 objects allocated from: > > If the preceding stack traces are not enough to find the leaks, try > running THIS shell command: > > pprof bro "/tmp/bro.3885.net_run-end.heap" --inuse_objects --lines > --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 --gv > > > > I can't tell what caused this error. Has anyone seen this before? This > is my first time running a memory leak test and this is happening for > all trace files, so I'm not sure if this is normal (guessing no). > > Josh > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150202/8158118c/attachment.html From jdonnelly at dyn.com Mon Feb 2 06:15:35 2015 From: jdonnelly at dyn.com (John Donnelly) Date: Mon, 2 Feb 2015 08:15:35 -0600 Subject: [Bro] Memory leak output In-Reply-To: References: Message-ID: When I use -m -M options with bro the output suggestions running "pprof" but the cli args don't work: pprof /opt/bro/bin/bro "/tmp/bro.17276.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 --gv pprof: invalid option -- '-' pprof: invalid option -- '-' pprof: invalid option -- '-' pprof: invalid option -- 'h' pprof: invalid option -- 'h' pprof: invalid option -- 'k' pprof: invalid option -- '-' pprof: invalid option -- 'g' pprof: invalid option -- '-' pprof: invalid option -- '-' pprof: invalid option -- 'g' usage: pprof [-c|-b|-m|-t|-e|-i|-v] [-r] [-s] [-n num] [-f filename] [-p] [-l] [-d] [node numbers] -a : Show all location information available -c : Sort according to number of Calls -b : Sort according to number of suBroutines called by a function -m : Sort according to Milliseconds (exclusive time total) -t : Sort according to Total milliseconds (inclusive time total) (default) -e : Sort according to Exclusive time per call (msec/call) -i : Sort according to Inclusive time per call (total msec/call) -v : Sort according to Standard Deviation (excl usec) -r : Reverse sorting order -s : print only Summary profile information -n : print only first number of functions -f filename : specify full path and Filename without node ids -p : suPpress conversion to hh:mm:ss:mmm format -l : List all functions and exit -d : Dump output format (for tau_reduce) [node numbers] : prints only info about all contexts/threads of given node numbers root at bro-x64-01:~# On Mon, Feb 2, 2015 at 6:57 AM, John Donnelly wrote: > I have built Bro with the perftools enabled and get similar messages ; the > pprof cli notice I've seen doesn't work either. > > On Sun, Feb 1, 2015 at 2:34 PM, Josh Liburdi > wrote: > >> Hey everyone, >> >> I'm performing testing for memory leaks and am running into an >> interesting perftools error message: >> >> Thread finding failed with -1 errno=1 >> Could not find thread stacks. Will likely report false leak positives. >> Have memory regions w/o callers: might report false leaks >> Leak check net_run detected leaks of 1520 bytes in 10 objects >> The 1 largest leaks: >> Leak of 1520 bytes in 10 objects allocated from: >> >> If the preceding stack traces are not enough to find the leaks, try >> running THIS shell command: >> >> pprof bro "/tmp/bro.3885.net_run-end.heap" --inuse_objects --lines >> --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 --gv >> >> >> >> I can't tell what caused this error. Has anyone seen this before? This >> is my first time running a memory leak test and this is happening for >> all trace files, so I'm not sure if this is normal (guessing no). >> >> Josh >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150202/0d88730e/attachment.html From seth at icir.org Mon Feb 2 07:53:22 2015 From: seth at icir.org (Seth Hall) Date: Mon, 2 Feb 2015 10:53:22 -0500 Subject: [Bro] Best practice on how to customize an officially distributed script In-Reply-To: References: Message-ID: > On Feb 2, 2015, at 12:35 AM, Luis Miguel Silva wrote: > > I would like to change the known-hosts.bro script to log both the ip and macaddr for all known hosts in my network. Are you collecting mac addresses from the DHCP analyzer? > What are the best practices for customizing scripts that ship with bro (e.g. distributed in the /usr/share/bro/* directory)? > Am I supposed to just: > - copy the script I want to customize to my share/bro/site/ > - and change local.bro to load the script in share/bro/site/ instead of share/bro/policy/protocols/conn/known-hosts.bro? That?s probably the best option. At the very least, if you?re loading the one out of your site directory you won?t have to worry about interfering with the one in the policy directory. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From luismiguelferreirasilva at gmail.com Mon Feb 2 08:49:45 2015 From: luismiguelferreirasilva at gmail.com (Luis Miguel Silva) Date: Mon, 2 Feb 2015 09:49:45 -0700 Subject: [Bro] Best practice on how to customize an officially distributed script In-Reply-To: References: Message-ID: I haven't given that much thought about how I'm going to capture the mac addr right now. :o) My first concern was to understand what are the best practices to customize an existing stock script. For instance, I don't know if it is possible to overload / extend other script's functions? If so, I'm interested in that, seeing as I do not want to replace / customize ALL script functionality. Originally, I had thought about running an arp query of some sort (maybe calling out an external script, which I'm guessing should be possible?) to figure out what the mac is for each local ip addr. Is there a more elegant / scalable way to do it? Thank you, Luis On Mon, Feb 2, 2015 at 8:53 AM, Seth Hall wrote: > > > On Feb 2, 2015, at 12:35 AM, Luis Miguel Silva < > luismiguelferreirasilva at gmail.com> wrote: > > > > I would like to change the known-hosts.bro script to log both the ip and > macaddr for all known hosts in my network. > > Are you collecting mac addresses from the DHCP analyzer? > > > What are the best practices for customizing scripts that ship with bro > (e.g. distributed in the /usr/share/bro/* directory)? > > Am I supposed to just: > > - copy the script I want to customize to my share/bro/site/ > > - and change local.bro to load the script in share/bro/site/ instead of > share/bro/policy/protocols/conn/known-hosts.bro? > > That?s probably the best option. At the very least, if you?re loading the > one out of your site directory you won?t have to worry about interfering > with the one in the policy directory. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150202/afcc0dc7/attachment-0001.html From luismiguelferreirasilva at gmail.com Mon Feb 2 08:51:34 2015 From: luismiguelferreirasilva at gmail.com (Luis Miguel Silva) Date: Mon, 2 Feb 2015 09:51:34 -0700 Subject: [Bro] Best practice on how to customize an officially distributed script In-Reply-To: References: Message-ID: ...by the way, I should have said this in my previous email... I do not think I can simply look at the DHCP info, seeing that some of the hosts in my network MIGHT have statically defined ip addresses. The known-hosts script looks at src and dest ip addrs to figure out who's out there, right? Thanks, Luis On Mon, Feb 2, 2015 at 9:49 AM, Luis Miguel Silva < luismiguelferreirasilva at gmail.com> wrote: > I haven't given that much thought about how I'm going to capture the mac > addr right now. :o) > My first concern was to understand what are the best practices to > customize an existing stock script. > > For instance, I don't know if it is possible to overload / extend other > script's functions? If so, I'm interested in that, seeing as I do not want > to replace / customize ALL script functionality. > > Originally, I had thought about running an arp query of some sort (maybe > calling out an external script, which I'm guessing should be possible?) to > figure out what the mac is for each local ip addr. Is there a more elegant > / scalable way to do it? > > Thank you, > Luis > > > On Mon, Feb 2, 2015 at 8:53 AM, Seth Hall wrote: > >> >> > On Feb 2, 2015, at 12:35 AM, Luis Miguel Silva < >> luismiguelferreirasilva at gmail.com> wrote: >> > >> > I would like to change the known-hosts.bro script to log both the ip >> and macaddr for all known hosts in my network. >> >> Are you collecting mac addresses from the DHCP analyzer? >> >> > What are the best practices for customizing scripts that ship with bro >> (e.g. distributed in the /usr/share/bro/* directory)? >> > Am I supposed to just: >> > - copy the script I want to customize to my share/bro/site/ >> > - and change local.bro to load the script in share/bro/site/ instead of >> share/bro/policy/protocols/conn/known-hosts.bro? >> >> That?s probably the best option. At the very least, if you?re loading >> the one out of your site directory you won?t have to worry about >> interfering with the one in the policy directory. >> >> .Seth >> >> -- >> Seth Hall >> International Computer Science Institute >> (Bro) because everyone has a network >> http://www.bro.org/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150202/367692f2/attachment.html From jsiwek at illinois.edu Mon Feb 2 09:32:06 2015 From: jsiwek at illinois.edu (Siwek, Jon) Date: Mon, 2 Feb 2015 17:32:06 +0000 Subject: [Bro] Memory leak output In-Reply-To: References: Message-ID: <79EF60DD-37A9-4331-8D39-7C7A9B6A8058@illinois.edu> > On Feb 1, 2015, at 2:34 PM, Josh Liburdi wrote: > > Hey everyone, > > I'm performing testing for memory leaks and am running into an > interesting perftools error message: > > Thread finding failed with -1 errno=1 > Could not find thread stacks. Will likely report false leak positives. > Have memory regions w/o callers: might report false leaks > Leak check net_run detected leaks of 1520 bytes in 10 objects > The 1 largest leaks: > Leak of 1520 bytes in 10 objects allocated from: > > If the preceding stack traces are not enough to find the leaks, try > running THIS shell command: > > pprof bro "/tmp/bro.3885.net_run-end.heap" --inuse_objects --lines > --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 --gv > > > > I can't tell what caused this error. Has anyone seen this before? Running the `pprof` command that it gives (I usually omit ?gv because I don?t have/want ghostview output) will help determine the cause. The `top` command in pprof usually gives me enough info to spot problems. > is my first time running a memory leak test and this is happening for > all trace files, so I'm not sure if this is normal (guessing no). No, not normal, but if you happened to be testing git/master there was a recent leak introduced; now fixed: https://github.com/bro/bro/commit/21c7642f6215960e1e9faf65d581e41dacb8de7c - Jon From jsiwek at illinois.edu Mon Feb 2 09:36:31 2015 From: jsiwek at illinois.edu (Siwek, Jon) Date: Mon, 2 Feb 2015 17:36:31 +0000 Subject: [Bro] Memory leak output In-Reply-To: References: Message-ID: <4B138781-CA8E-4051-A6D1-27B46E9B4830@illinois.edu> > On Feb 2, 2015, at 8:15 AM, John Donnelly wrote: > > When I use -m -M options with bro the output suggestions running "pprof" but the cli args don't work: Instead of ?pprof?, which is sometimes associated w/ a package called TAU, check if you have something named "google-pprof?, which will be the actual tool associated w/ gperftools. - Jon From liburdi.joshua at gmail.com Mon Feb 2 12:07:59 2015 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Mon, 2 Feb 2015 12:07:59 -0800 Subject: [Bro] Memory leak output In-Reply-To: <79EF60DD-37A9-4331-8D39-7C7A9B6A8058@illinois.edu> References: <79EF60DD-37A9-4331-8D39-7C7A9B6A8058@illinois.edu> Message-ID: Jon, Thanks for the reply. This particular Bro package was the stable 2.3.2 release with an additional TCP analyzer I am testing, but perftools reports leaks with the stable release of 2.3.2 on its own. Some additional data for you if you're interested: Using a 1.7GB pcap file ... perftools reported Bro 2.3.2 as having a leak of 1216 bytes in 8 objects perftools reported Bro 2.3.2 plus my analyzer as having a leak of 1368 bytes in 9 objects perftools reported git/master as having a leak of 5136 bytes in 78 objects Running the top command in pprof for all runs resulted in 0.0MB shown. All runs did not indicate where the objects were allocated from. I also ran all packages (standard 2.3.2, 2.3.2 with my analyzer, and git/master) through valgrind and that found zero leaks in all runs. Do you have an opinion on which tool (perftools or valgrind) is more "accurate" with regard to reporting leaks? I'm not sure why perftools is not finding thread stacks (unless they simply don't exist and the reported leaks are false positives) ... Josh On Mon, Feb 2, 2015 at 9:32 AM, Siwek, Jon wrote: > >> On Feb 1, 2015, at 2:34 PM, Josh Liburdi wrote: >> >> Hey everyone, >> >> I'm performing testing for memory leaks and am running into an >> interesting perftools error message: >> >> Thread finding failed with -1 errno=1 >> Could not find thread stacks. Will likely report false leak positives. >> Have memory regions w/o callers: might report false leaks >> Leak check net_run detected leaks of 1520 bytes in 10 objects >> The 1 largest leaks: >> Leak of 1520 bytes in 10 objects allocated from: >> >> If the preceding stack traces are not enough to find the leaks, try >> running THIS shell command: >> >> pprof bro "/tmp/bro.3885.net_run-end.heap" --inuse_objects --lines >> --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 --gv >> >> >> >> I can't tell what caused this error. Has anyone seen this before? > > Running the `pprof` command that it gives (I usually omit ?gv because I don?t have/want ghostview output) will help determine the cause. The `top` command in pprof usually gives me enough info to spot problems. > >> is my first time running a memory leak test and this is happening for >> all trace files, so I'm not sure if this is normal (guessing no). > > No, not normal, but if you happened to be testing git/master there was a recent leak introduced; now fixed: > > https://github.com/bro/bro/commit/21c7642f6215960e1e9faf65d581e41dacb8de7c > > - Jon From jsiwek at illinois.edu Mon Feb 2 12:41:30 2015 From: jsiwek at illinois.edu (Siwek, Jon) Date: Mon, 2 Feb 2015 20:41:30 +0000 Subject: [Bro] Memory leak output In-Reply-To: References: <79EF60DD-37A9-4331-8D39-7C7A9B6A8058@illinois.edu> Message-ID: <89C96735-B3DC-48DF-91E5-EE82EBDAC71F@illinois.edu> > On Feb 2, 2015, at 2:07 PM, Josh Liburdi wrote: > > I also ran all packages (standard 2.3.2, 2.3.2 with my analyzer, and > git/master) through valgrind and that found zero leaks in all runs. Do > you have an opinion on which tool (perftools or valgrind) is more > "accurate" with regard to reporting leaks? I'm not sure why perftools > is not finding thread stacks (unless they simply don't exist and the > reported leaks are false positives) ... How did you configure/build Bro? My leak check configuration is ./configure --enable-debug --enable-perftools-debug Regarding accuracy, I haven?t noticed much difference between valgrind and gperftools. Sometimes I prefer valgrind just to also check for other memory errors (e.g. invalid read/write, using uninitialized). AddressSanitizer is also a nice tool. However, unless you?re using a suppression file, valgrind should be reporting a lot of ?leaks? in Bro?s initialization routines that aren?t really concerning. With gperftools, Bro has been instrumented to only check for leaks in the main I/O loop, so leaks it reports are usually always concerning. - Jon From liburdi.joshua at gmail.com Mon Feb 2 13:25:32 2015 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Mon, 2 Feb 2015 13:25:32 -0800 Subject: [Bro] Memory leak output In-Reply-To: <89C96735-B3DC-48DF-91E5-EE82EBDAC71F@illinois.edu> References: <79EF60DD-37A9-4331-8D39-7C7A9B6A8058@illinois.edu> <89C96735-B3DC-48DF-91E5-EE82EBDAC71F@illinois.edu> Message-ID: That's odd, I am using the configuration referenced on the Finding Memory Leaks page: ./configure --enable-debug --enable-perftools --enable-perftools-debug I tried your configuration as well and receive the same results (gperftools reports memory leaks but can't find thread stacks, valgrind finds no memory leaks whatsoever). There must be something wrong with one of my installations. On Mon, Feb 2, 2015 at 12:41 PM, Siwek, Jon wrote: > >> On Feb 2, 2015, at 2:07 PM, Josh Liburdi wrote: >> >> I also ran all packages (standard 2.3.2, 2.3.2 with my analyzer, and >> git/master) through valgrind and that found zero leaks in all runs. Do >> you have an opinion on which tool (perftools or valgrind) is more >> "accurate" with regard to reporting leaks? I'm not sure why perftools >> is not finding thread stacks (unless they simply don't exist and the >> reported leaks are false positives) ... > > How did you configure/build Bro? My leak check configuration is > > ./configure --enable-debug --enable-perftools-debug > > Regarding accuracy, I haven?t noticed much difference between valgrind and gperftools. Sometimes I prefer valgrind just to also check for other memory errors (e.g. invalid read/write, using uninitialized). AddressSanitizer is also a nice tool. > > However, unless you?re using a suppression file, valgrind should be reporting a lot of ?leaks? in Bro?s initialization routines that aren?t really concerning. With gperftools, Bro has been instrumented to only check for leaks in the main I/O loop, so leaks it reports are usually always concerning. > > - Jon From liburdi.joshua at gmail.com Mon Feb 2 13:26:38 2015 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Mon, 2 Feb 2015 13:26:38 -0800 Subject: [Bro] Memory leak output In-Reply-To: References: <79EF60DD-37A9-4331-8D39-7C7A9B6A8058@illinois.edu> <89C96735-B3DC-48DF-91E5-EE82EBDAC71F@illinois.edu> Message-ID: Addtionally, my Bro debug.log is empty. On Mon, Feb 2, 2015 at 1:25 PM, Josh Liburdi wrote: > That's odd, I am using the configuration referenced on the Finding > Memory Leaks page: ./configure --enable-debug --enable-perftools > --enable-perftools-debug > > I tried your configuration as well and receive the same results > (gperftools reports memory leaks but can't find thread stacks, > valgrind finds no memory leaks whatsoever). There must be something > wrong with one of my installations. > > On Mon, Feb 2, 2015 at 12:41 PM, Siwek, Jon wrote: >> >>> On Feb 2, 2015, at 2:07 PM, Josh Liburdi wrote: >>> >>> I also ran all packages (standard 2.3.2, 2.3.2 with my analyzer, and >>> git/master) through valgrind and that found zero leaks in all runs. Do >>> you have an opinion on which tool (perftools or valgrind) is more >>> "accurate" with regard to reporting leaks? I'm not sure why perftools >>> is not finding thread stacks (unless they simply don't exist and the >>> reported leaks are false positives) ... >> >> How did you configure/build Bro? My leak check configuration is >> >> ./configure --enable-debug --enable-perftools-debug >> >> Regarding accuracy, I haven?t noticed much difference between valgrind and gperftools. Sometimes I prefer valgrind just to also check for other memory errors (e.g. invalid read/write, using uninitialized). AddressSanitizer is also a nice tool. >> >> However, unless you?re using a suppression file, valgrind should be reporting a lot of ?leaks? in Bro?s initialization routines that aren?t really concerning. With gperftools, Bro has been instrumented to only check for leaks in the main I/O loop, so leaks it reports are usually always concerning. >> >> - Jon From jsiwek at illinois.edu Mon Feb 2 14:52:13 2015 From: jsiwek at illinois.edu (Siwek, Jon) Date: Mon, 2 Feb 2015 22:52:13 +0000 Subject: [Bro] Memory leak output In-Reply-To: References: <79EF60DD-37A9-4331-8D39-7C7A9B6A8058@illinois.edu> <89C96735-B3DC-48DF-91E5-EE82EBDAC71F@illinois.edu> Message-ID: <444E2B73-E1A6-4AB3-9E11-79BD30F23B99@illinois.edu> > On Feb 2, 2015, at 3:26 PM, Josh Liburdi wrote: > > Addtionally, my Bro debug.log is empty. An empty debug.log is fine. It only has contents if at least one of the various debug streams is enabled via a -B flag when running bro. DebugLogger::streams in src/DebugLogger.cc has a list of stream names. > On Mon, Feb 2, 2015 at 1:25 PM, Josh Liburdi wrote: >> That's odd, I am using the configuration referenced on the Finding >> Memory Leaks page: ./configure --enable-debug --enable-perftools >> --enable-perftools-debug >> >> I tried your configuration as well and receive the same results >> (gperftools reports memory leaks but can't find thread stacks, >> valgrind finds no memory leaks whatsoever). There must be something >> wrong with one of my installations. For valgrind, maybe check that ~/.valgrindrc, $VALGRIND_OPTS, ./.valgrindrc aren?t doing something to change leak-check behavior and make sure to do ?leak-check=full. For either pprof or valgrind, maybe make sure the bro binary is the one you expect (i.e. use a full path) and that it?s not a script or other program that just exec?s bro. Otherwise, maybe you?ll have to start troubleshooting from a simple toy program that you?ve written and know always leaks memory. - Jon From liburdi.joshua at gmail.com Mon Feb 2 17:01:39 2015 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Mon, 2 Feb 2015 17:01:39 -0800 Subject: [Bro] Memory leak output In-Reply-To: <444E2B73-E1A6-4AB3-9E11-79BD30F23B99@illinois.edu> References: <79EF60DD-37A9-4331-8D39-7C7A9B6A8058@illinois.edu> <89C96735-B3DC-48DF-91E5-EE82EBDAC71F@illinois.edu> <444E2B73-E1A6-4AB3-9E11-79BD30F23B99@illinois.edu> Message-ID: Thanks Jon. I have a simple program that definitely leaks memory and valgrind reports this accurately, but I still cannot get it to work with Bro. If you have time, would you copy / paste the command line arguments you use to test Bro with valgrind? On Mon, Feb 2, 2015 at 2:52 PM, Siwek, Jon wrote: > >> On Feb 2, 2015, at 3:26 PM, Josh Liburdi wrote: >> >> Addtionally, my Bro debug.log is empty. > > An empty debug.log is fine. It only has contents if at least one of the various debug streams is enabled via a -B flag when running bro. DebugLogger::streams in src/DebugLogger.cc has a list of stream names. > >> On Mon, Feb 2, 2015 at 1:25 PM, Josh Liburdi wrote: >>> That's odd, I am using the configuration referenced on the Finding >>> Memory Leaks page: ./configure --enable-debug --enable-perftools >>> --enable-perftools-debug >>> >>> I tried your configuration as well and receive the same results >>> (gperftools reports memory leaks but can't find thread stacks, >>> valgrind finds no memory leaks whatsoever). There must be something >>> wrong with one of my installations. > > For valgrind, maybe check that ~/.valgrindrc, $VALGRIND_OPTS, ./.valgrindrc aren?t doing something to change leak-check behavior and make sure to do ?leak-check=full. > > For either pprof or valgrind, maybe make sure the bro binary is the one you expect (i.e. use a full path) and that it?s not a script or other program that just exec?s bro. > > Otherwise, maybe you?ll have to start troubleshooting from a simple toy program that you?ve written and know always leaks memory. > > - Jon From jdonnelly at dyn.com Mon Feb 2 17:27:16 2015 From: jdonnelly at dyn.com (John Donnelly) Date: Mon, 2 Feb 2015 19:27:16 -0600 Subject: [Bro] Memory leak output In-Reply-To: References: <79EF60DD-37A9-4331-8D39-7C7A9B6A8058@illinois.edu> <89C96735-B3DC-48DF-91E5-EE82EBDAC71F@illinois.edu> <444E2B73-E1A6-4AB3-9E11-79BD30F23B99@illinois.edu> Message-ID: valgrind --log-file=val.txt --tool=memcheck --leak-check=full -v /opt/bro/bin/bro -i lo -i eth0 -i eth1 -b -C /opt/bro/share/bro/base/protocols/dns/main.bro The results are saved in "val.txt" I haven't seen any reports of leaks . On Mon, Feb 2, 2015 at 7:01 PM, Josh Liburdi wrote: > Thanks Jon. I have a simple program that definitely leaks memory and > valgrind reports this accurately, but I still cannot get it to work > with Bro. If you have time, would you copy / paste the command line > arguments you use to test Bro with valgrind? > > On Mon, Feb 2, 2015 at 2:52 PM, Siwek, Jon wrote: > > > >> On Feb 2, 2015, at 3:26 PM, Josh Liburdi > wrote: > >> > >> Addtionally, my Bro debug.log is empty. > > > > An empty debug.log is fine. It only has contents if at least one of the > various debug streams is enabled via a -B flag when running bro. > DebugLogger::streams in src/DebugLogger.cc has a list of stream names. > > > >> On Mon, Feb 2, 2015 at 1:25 PM, Josh Liburdi > wrote: > >>> That's odd, I am using the configuration referenced on the Finding > >>> Memory Leaks page: ./configure --enable-debug --enable-perftools > >>> --enable-perftools-debug > >>> > >>> I tried your configuration as well and receive the same results > >>> (gperftools reports memory leaks but can't find thread stacks, > >>> valgrind finds no memory leaks whatsoever). There must be something > >>> wrong with one of my installations. > > > > For valgrind, maybe check that ~/.valgrindrc, $VALGRIND_OPTS, > ./.valgrindrc aren?t doing something to change leak-check behavior and make > sure to do ?leak-check=full. > > > > For either pprof or valgrind, maybe make sure the bro binary is the one > you expect (i.e. use a full path) and that it?s not a script or other > program that just exec?s bro. > > > > Otherwise, maybe you?ll have to start troubleshooting from a simple toy > program that you?ve written and know always leaks memory. > > > > - Jon > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150202/2371abfd/attachment-0001.html From liburdi.joshua at gmail.com Mon Feb 2 19:07:47 2015 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Mon, 02 Feb 2015 19:07:47 -0800 (PST) Subject: [Bro] Memory leak output In-Reply-To: References: Message-ID: <1422932867394.d850f489@Nodemailer> Same for me, but at the least it should be reporting leaks in the init routines. Since I'm working on an analyzer I'm pretty determined to get this working accurately ...? Josh On Monday, Feb 2, 2015 at 5:27 PM, John Donnelly , wrote: valgrind --log-file=val.txt --tool=memcheck --leak-check=full -v /opt/bro/bin/bro -i lo -i eth0 -i eth1 -b -C /opt/bro/share/bro/base/protocols/dns/main.bro? The results are saved in "val.txt"? I haven't seen any reports of leaks .? On Mon, Feb 2, 2015 at 7:01 PM, Josh Liburdi wrote: Thanks Jon. I have a simple program that definitely leaks memory and valgrind reports this accurately, but I still cannot get it to work with Bro. If you have time, would you copy / paste the command line arguments you use to test Bro with valgrind? On Mon, Feb 2, 2015 at 2:52 PM, Siwek, Jon wrote: > >> On Feb 2, 2015, at 3:26 PM, Josh Liburdi wrote: >> >> Addtionally, my Bro debug.log is empty. > > An empty debug.log is fine.? It only has contents if at least one of the various debug streams is enabled via a -B flag when running bro.? DebugLogger::streams in src/DebugLogger.cc has a list of stream names. > >> On Mon, Feb 2, 2015 at 1:25 PM, Josh Liburdi wrote: >>> That's odd, I am using the configuration referenced on the Finding >>> Memory Leaks page: ./configure --enable-debug --enable-perftools >>> --enable-perftools-debug >>> >>> I tried your configuration as well and receive the same results >>> (gperftools reports memory leaks but can't find thread stacks, >>> valgrind finds no memory leaks whatsoever). There must be something >>> wrong with one of my installations. > > For valgrind, maybe check that ~/.valgrindrc, $VALGRIND_OPTS, ./.valgrindrc aren?t doing something to change leak-check behavior and make sure to do ?leak-check=full. > > For either pprof or valgrind, maybe make sure the bro binary is the one you expect (i.e. use a full path) and that it?s not a script or other program that just exec?s bro. > > Otherwise, maybe you?ll have to start troubleshooting from a simple toy program that you?ve written and know always leaks memory. > > - Jon _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150202/caed9b1f/attachment.html From jsiwek at illinois.edu Tue Feb 3 08:12:55 2015 From: jsiwek at illinois.edu (Siwek, Jon) Date: Tue, 3 Feb 2015 16:12:55 +0000 Subject: [Bro] Memory leak output In-Reply-To: References: <79EF60DD-37A9-4331-8D39-7C7A9B6A8058@illinois.edu> <89C96735-B3DC-48DF-91E5-EE82EBDAC71F@illinois.edu> <444E2B73-E1A6-4AB3-9E11-79BD30F23B99@illinois.edu> Message-ID: <7CACCF9A-D369-4208-BCD3-1B0FD606E2D3@illinois.edu> > On Feb 2, 2015, at 7:01 PM, Josh Liburdi wrote: > > Thanks Jon. I have a simple program that definitely leaks memory and > valgrind reports this accurately, but I still cannot get it to work > with Bro. If you have time, would you copy / paste the command line > arguments you use to test Bro with valgrind? git clone --recursive git://git.bro.org/bro bro-tmp cd bro-tmp ./configure --enable-debug --disable-perftools cd build/ make -j8 . bro-path-dev.sh valgrind --leak-check=yes ./src/bro -r ../testing/btest/Traces/http/get.trace ==12109== Memcheck, a memory error detector ==12109== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al. ==12109== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info ==12109== Command: ./src/bro -r ../testing/btest/Traces/http/get.trace ==12109== ==12109== ==12109== HEAP SUMMARY: ==12109== in use at exit: 30,275,602 bytes in 434,261 blocks ==12109== total heap usage: 1,340,380 allocs, 906,119 frees, 115,312,199 bytes allocated ==12109== ==12109== 1 bytes in 1 blocks are definitely lost in loss record 2 of 5,455 ==12109== at 0x4A06A2E: malloc (vg_replace_malloc.c:270) ==12109== by 0x3FB6080E91: strdup (in /lib64/libc-2.12.so) ==12109== by 0x6D3585: main (main.cc:533) ? - Jon From liburdi.joshua at gmail.com Tue Feb 3 09:43:08 2015 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Tue, 3 Feb 2015 09:43:08 -0800 Subject: [Bro] Memory leak output In-Reply-To: <7CACCF9A-D369-4208-BCD3-1B0FD606E2D3@illinois.edu> References: <79EF60DD-37A9-4331-8D39-7C7A9B6A8058@illinois.edu> <89C96735-B3DC-48DF-91E5-EE82EBDAC71F@illinois.edu> <444E2B73-E1A6-4AB3-9E11-79BD30F23B99@illinois.edu> <7CACCF9A-D369-4208-BCD3-1B0FD606E2D3@illinois.edu> Message-ID: Wow, thanks Jon-- that helped a lot. When you test with perftools, do you do anything different aside from change the configure options (--enable-debug --enable-perftools-debug) and run without valgrind? On Tue, Feb 3, 2015 at 8:12 AM, Siwek, Jon wrote: > >> On Feb 2, 2015, at 7:01 PM, Josh Liburdi wrote: >> >> Thanks Jon. I have a simple program that definitely leaks memory and >> valgrind reports this accurately, but I still cannot get it to work >> with Bro. If you have time, would you copy / paste the command line >> arguments you use to test Bro with valgrind? > > git clone --recursive git://git.bro.org/bro bro-tmp > cd bro-tmp > ./configure --enable-debug --disable-perftools > cd build/ > make -j8 > . bro-path-dev.sh > valgrind --leak-check=yes ./src/bro -r ../testing/btest/Traces/http/get.trace > ==12109== Memcheck, a memory error detector > ==12109== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al. > ==12109== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info > ==12109== Command: ./src/bro -r ../testing/btest/Traces/http/get.trace > ==12109== > ==12109== > ==12109== HEAP SUMMARY: > ==12109== in use at exit: 30,275,602 bytes in 434,261 blocks > ==12109== total heap usage: 1,340,380 allocs, 906,119 frees, 115,312,199 bytes allocated > ==12109== > ==12109== 1 bytes in 1 blocks are definitely lost in loss record 2 of 5,455 > ==12109== at 0x4A06A2E: malloc (vg_replace_malloc.c:270) > ==12109== by 0x3FB6080E91: strdup (in /lib64/libc-2.12.so) > ==12109== by 0x6D3585: main (main.cc:533) > ? > > - Jon From jsiwek at illinois.edu Tue Feb 3 10:17:16 2015 From: jsiwek at illinois.edu (Siwek, Jon) Date: Tue, 3 Feb 2015 18:17:16 +0000 Subject: [Bro] Memory leak output In-Reply-To: References: <79EF60DD-37A9-4331-8D39-7C7A9B6A8058@illinois.edu> <89C96735-B3DC-48DF-91E5-EE82EBDAC71F@illinois.edu> <444E2B73-E1A6-4AB3-9E11-79BD30F23B99@illinois.edu> <7CACCF9A-D369-4208-BCD3-1B0FD606E2D3@illinois.edu> Message-ID: > On Feb 3, 2015, at 11:43 AM, Josh Liburdi wrote: > > Wow, thanks Jon-- that helped a lot. When you test with perftools, do > you do anything different aside from change the configure options > (--enable-debug --enable-perftools-debug) and run without valgrind? Here?s an example w/ gperftools: $ git clone --recursive git://git.bro.org/bro bro-tmp $ cd bro-tmp $ ./configure --enable-debug --enable-perftools-debug $ cd build/ $ vim ../src/Net.cc $ git diff diff --git a/src/Net.cc b/src/Net.cc index adac9c0..53169d2 100644 --- a/src/Net.cc +++ b/src/Net.cc @@ -297,6 +297,7 @@ void net_run() while ( iosource_mgr->Size() || (BifConst::exit_only_after_terminate && ! terminating) ) { + char* leak = new char; double ts; iosource::IOSource* src = iosource_mgr->FindSoonest(&ts); $ make -j8 $ . bro-path-dev.sh $ HEAP_CHECK_DUMP_DIRECTORY=. HEAPCHECK=local ./src/bro -m -r ../testing/btest/Traces/http/get.trace WARNING: Perftools heap leak checker is active -- Performance may suffer Have memory regions w/o callers: might report false leaks Leak check net_run detected leaks of 23 bytes in 23 objects The 1 largest leaks: Leak of 23 bytes in 23 objects allocated from: @ 785d25 @ 6d61d2 @ 3fb601ed5d @ 6b2359 If the preceding stack traces are not enough to find the leaks, try running THIS shell command: pprof ./src/bro "./bro.25523.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 --gv If you are still puzzled about why the leaks are there, try rerunning this program with HEAP_CHECK_TEST_POINTER_ALIGNMENT=1 and/or with HEAP_CHECK_MAX_POINTER_OFFSET=-1 If the leak report occurs in a small fraction of runs, try running with TCMALLOC_MAX_FREE_QUEUE_SIZE of few hundred MB or with TCMALLOC_RECLAIM_MEMORY=false, it might help find leaks more repeatably Memory leaks - aborting. Aborted (core dumped) $ pprof ./src/bro "./bro.25523.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 Using local file ./src/bro. Using local file ./bro.25523.net_run-end.heap. Welcome to pprof! For help, type 'help'. (pprof) top Total: 23 objects 23 100.0% 100.0% 23 100.0% net_run /home/jsiwek/bro-tmp/src/Net.cc:300 0 0.0% 100.0% 23 100.0% __libc_start_main ??:0 0 0.0% 100.0% 23 100.0% _start ??:0 0 0.0% 100.0% 23 100.0% main /home/jsiwek/bro-tmp/src/main.cc:1200 (pprof) quit From liburdi.joshua at gmail.com Tue Feb 3 10:53:23 2015 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Tue, 3 Feb 2015 10:53:23 -0800 Subject: [Bro] Memory leak output In-Reply-To: References: <79EF60DD-37A9-4331-8D39-7C7A9B6A8058@illinois.edu> <89C96735-B3DC-48DF-91E5-EE82EBDAC71F@illinois.edu> <444E2B73-E1A6-4AB3-9E11-79BD30F23B99@illinois.edu> <7CACCF9A-D369-4208-BCD3-1B0FD606E2D3@illinois.edu> Message-ID: Thanks ... I ran the same commands (also edited the Net.cc file) and received this output: WARNING: Perftools heap leak checker is active -- Performance may suffer Thread finding failed with -1 errno=1 Could not find thread stacks. Will likely report false leak positives. Have memory regions w/o callers: might report false leaks Leak check net_run detected leaks of 475 bytes in 22 objects The 1 largest leaks: Leak of 475 bytes in 22 objects allocated from: If the preceding stack traces are not enough to find the leaks, try running THIS shell command: pprof ./src/bro "./bro.4119.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 --gv Something must be off in my gperftools config, not sure what since I installed the standard package. Thanks, Josh On Tue, Feb 3, 2015 at 10:17 AM, Siwek, Jon wrote: > >> On Feb 3, 2015, at 11:43 AM, Josh Liburdi wrote: >> >> Wow, thanks Jon-- that helped a lot. When you test with perftools, do >> you do anything different aside from change the configure options >> (--enable-debug --enable-perftools-debug) and run without valgrind? > > Here?s an example w/ gperftools: > > $ git clone --recursive git://git.bro.org/bro bro-tmp > $ cd bro-tmp > $ ./configure --enable-debug --enable-perftools-debug > $ cd build/ > $ vim ../src/Net.cc > > $ git diff > diff --git a/src/Net.cc b/src/Net.cc > index adac9c0..53169d2 100644 > --- a/src/Net.cc > +++ b/src/Net.cc > @@ -297,6 +297,7 @@ void net_run() > while ( iosource_mgr->Size() || > (BifConst::exit_only_after_terminate && ! terminating) ) > { > + char* leak = new char; > double ts; > iosource::IOSource* src = iosource_mgr->FindSoonest(&ts); > > $ make -j8 > $ . bro-path-dev.sh > $ HEAP_CHECK_DUMP_DIRECTORY=. HEAPCHECK=local ./src/bro -m -r ../testing/btest/Traces/http/get.trace > > WARNING: Perftools heap leak checker is active -- Performance may suffer > Have memory regions w/o callers: might report false leaks > Leak check net_run detected leaks of 23 bytes in 23 objects > The 1 largest leaks: > Leak of 23 bytes in 23 objects allocated from: > @ 785d25 > @ 6d61d2 > @ 3fb601ed5d > @ 6b2359 > > If the preceding stack traces are not enough to find the leaks, try running THIS shell command: > > pprof ./src/bro "./bro.25523.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 --gv > > If you are still puzzled about why the leaks are there, try rerunning this program with HEAP_CHECK_TEST_POINTER_ALIGNMENT=1 and/or with HEAP_CHECK_MAX_POINTER_OFFSET=-1 > If the leak report occurs in a small fraction of runs, try running with TCMALLOC_MAX_FREE_QUEUE_SIZE of few hundred MB or with TCMALLOC_RECLAIM_MEMORY=false, it might help find leaks more repeatably > Memory leaks - aborting. > Aborted (core dumped) > > $ pprof ./src/bro "./bro.25523.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 > > Using local file ./src/bro. > Using local file ./bro.25523.net_run-end.heap. > Welcome to pprof! For help, type 'help'. > (pprof) top > Total: 23 objects > 23 100.0% 100.0% 23 100.0% net_run /home/jsiwek/bro-tmp/src/Net.cc:300 > 0 0.0% 100.0% 23 100.0% __libc_start_main ??:0 > 0 0.0% 100.0% 23 100.0% _start ??:0 > 0 0.0% 100.0% 23 100.0% main /home/jsiwek/bro-tmp/src/main.cc:1200 > (pprof) quit > > From liburdi.joshua at gmail.com Tue Feb 3 11:11:41 2015 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Tue, 3 Feb 2015 11:11:41 -0800 Subject: [Bro] Memory leak output In-Reply-To: References: <79EF60DD-37A9-4331-8D39-7C7A9B6A8058@illinois.edu> <89C96735-B3DC-48DF-91E5-EE82EBDAC71F@illinois.edu> <444E2B73-E1A6-4AB3-9E11-79BD30F23B99@illinois.edu> <7CACCF9A-D369-4208-BCD3-1B0FD606E2D3@illinois.edu> Message-ID: Disregard my previous message ... I re-installed gperftools and it seems to be working now. Thanks a lot for your help, you've saved me (and hopefully others) lots of time and effort. Josh On Tue, Feb 3, 2015 at 10:53 AM, Josh Liburdi wrote: > Thanks ... I ran the same commands (also edited the Net.cc file) and > received this output: > > WARNING: Perftools heap leak checker is active -- Performance may suffer > Thread finding failed with -1 errno=1 > Could not find thread stacks. Will likely report false leak positives. > Have memory regions w/o callers: might report false leaks > Leak check net_run detected leaks of 475 bytes in 22 objects > The 1 largest leaks: > Leak of 475 bytes in 22 objects allocated from: > > > If the preceding stack traces are not enough to find the leaks, try > running THIS shell command: > > pprof ./src/bro "./bro.4119.net_run-end.heap" --inuse_objects --lines > --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 --gv > > > Something must be off in my gperftools config, not sure what since I > installed the standard package. > > Thanks, > Josh > > > On Tue, Feb 3, 2015 at 10:17 AM, Siwek, Jon wrote: >> >>> On Feb 3, 2015, at 11:43 AM, Josh Liburdi wrote: >>> >>> Wow, thanks Jon-- that helped a lot. When you test with perftools, do >>> you do anything different aside from change the configure options >>> (--enable-debug --enable-perftools-debug) and run without valgrind? >> >> Here?s an example w/ gperftools: >> >> $ git clone --recursive git://git.bro.org/bro bro-tmp >> $ cd bro-tmp >> $ ./configure --enable-debug --enable-perftools-debug >> $ cd build/ >> $ vim ../src/Net.cc >> >> $ git diff >> diff --git a/src/Net.cc b/src/Net.cc >> index adac9c0..53169d2 100644 >> --- a/src/Net.cc >> +++ b/src/Net.cc >> @@ -297,6 +297,7 @@ void net_run() >> while ( iosource_mgr->Size() || >> (BifConst::exit_only_after_terminate && ! terminating) ) >> { >> + char* leak = new char; >> double ts; >> iosource::IOSource* src = iosource_mgr->FindSoonest(&ts); >> >> $ make -j8 >> $ . bro-path-dev.sh >> $ HEAP_CHECK_DUMP_DIRECTORY=. HEAPCHECK=local ./src/bro -m -r ../testing/btest/Traces/http/get.trace >> >> WARNING: Perftools heap leak checker is active -- Performance may suffer >> Have memory regions w/o callers: might report false leaks >> Leak check net_run detected leaks of 23 bytes in 23 objects >> The 1 largest leaks: >> Leak of 23 bytes in 23 objects allocated from: >> @ 785d25 >> @ 6d61d2 >> @ 3fb601ed5d >> @ 6b2359 >> >> If the preceding stack traces are not enough to find the leaks, try running THIS shell command: >> >> pprof ./src/bro "./bro.25523.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 --gv >> >> If you are still puzzled about why the leaks are there, try rerunning this program with HEAP_CHECK_TEST_POINTER_ALIGNMENT=1 and/or with HEAP_CHECK_MAX_POINTER_OFFSET=-1 >> If the leak report occurs in a small fraction of runs, try running with TCMALLOC_MAX_FREE_QUEUE_SIZE of few hundred MB or with TCMALLOC_RECLAIM_MEMORY=false, it might help find leaks more repeatably >> Memory leaks - aborting. >> Aborted (core dumped) >> >> $ pprof ./src/bro "./bro.25523.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 >> >> Using local file ./src/bro. >> Using local file ./bro.25523.net_run-end.heap. >> Welcome to pprof! For help, type 'help'. >> (pprof) top >> Total: 23 objects >> 23 100.0% 100.0% 23 100.0% net_run /home/jsiwek/bro-tmp/src/Net.cc:300 >> 0 0.0% 100.0% 23 100.0% __libc_start_main ??:0 >> 0 0.0% 100.0% 23 100.0% _start ??:0 >> 0 0.0% 100.0% 23 100.0% main /home/jsiwek/bro-tmp/src/main.cc:1200 >> (pprof) quit >> >> From shasubra1 at gmail.com Tue Feb 3 16:21:06 2015 From: shasubra1 at gmail.com (shasubra1 at gmail.com) Date: Tue, 3 Feb 2015 16:21:06 -0800 Subject: [Bro] Sending logs to remote public cloud entity Message-ID: <05419566-D78C-4A9D-A422-D5B76F2B5A02@gmail.com> I am looking into setting up a Bro manager in AWS cloud, which will receive logs from multiple Bro workers on premise. I then plan to take the logs received on the manager and load them into my database on AWS. The Bro manager itself is merely to receive the logs and does not generate logs of its own. I read some documentation about Broccoli whereby I can configure an SSL tunnel by furnishing the manager with the a public cert, key and CA. I have not found much documentation nor discussion on this kind of a setup usage. I am wondering: - is this the recommended approach to send logs to a remote public cloud entity - the alternative is to send syslog?s but then I would need to set up stunnel or some other encrypted tunneling, which instead I am hoping to leverage the Broccoli SSL tunneling functionality - will the Bro manager scale to receive logs from multiple workers (like 10) - I can work around this by running multiple Bro managers listening on different ports - will the logs be written into the normal place on disk with the default writer Thanks in advance for your input. Shankar -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150203/52245868/attachment.html From just2 at arcor.de Wed Feb 4 04:09:10 2015 From: just2 at arcor.de (just2 at arcor.de) Date: Wed, 4 Feb 2015 13:09:10 +0100 (CET) Subject: [Bro] Bro Signature Framework Examples References: <652454446.113266.1423050443162.JavaMail.ngmail@webmail20.arcor-online.net> Message-ID: <884864139.113789.1423051750790.JavaMail.ngmail@webmail20.arcor-online.net> Hello everyone, for testing purposes, I want to run Bro with signatures (similar to Snort). On https://www.bro.org/sphinx/frameworks/signatures.html it is described how to configure bro to use a signature file. However, I did not find a sample signature file. Also, it is stated that Snort signatures can no longer be transfered to Bro. Is there another way to (easily) import a bulk of the most common signatures? Is there any example file? Thanks, Myra From liam.randall at gigaco.com Wed Feb 4 05:52:38 2015 From: liam.randall at gigaco.com (Liam Randall) Date: Wed, 4 Feb 2015 08:52:38 -0500 Subject: [Bro] Bro Signature Framework Examples In-Reply-To: <884864139.113789.1423051750790.JavaMail.ngmail@webmail20.arcor-online.net> References: <652454446.113266.1423050443162.JavaMail.ngmail@webmail20.arcor-online.net> <884864139.113789.1423051750790.JavaMail.ngmail@webmail20.arcor-online.net> Message-ID: Myra, If you look under policy/frameworks/signatures/detect-windows-shells.sig: https://github.com/bro/bro/blob/master/scripts/policy/frameworks/signatures/detect-windows-shells.sig You'll find an example signature that ships with Bro. Additionally, each protocol analyzer is enabled by a signature used in the dynamic protocol detection (dpd) process; for example please see http's signature: https://github.com/bro/bro/blob/master/scripts/base/protocols/http/dpd.sig There are a lot of novel uses of signatures in Bro; in Jon bitcoin mining protocol detection he uses a signature to enable an analysis process: https://github.com/jsiwek/bro_bitcoin Many of the "signatures" you would use to find basic indicators of compromise (domains, ip addresses, file hashes, etc) are handled by the intelligence framework: https://www.bro.org/sphinx/frameworks/intel.html V/r, Liam Randall On Wed, Feb 4, 2015 at 7:09 AM, wrote: > Hello everyone, > for testing purposes, I want to run Bro with signatures (similar to Snort). > On https://www.bro.org/sphinx/frameworks/signatures.html it is described > how to configure bro to use a signature file. > However, I did not find a sample signature file. Also, it is stated that > Snort signatures can no longer be transfered to Bro. > Is there another way to (easily) import a bulk of the most common > signatures? Is there any example file? > Thanks, > Myra > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150204/c82f7d54/attachment.html From seth at icir.org Wed Feb 4 06:44:07 2015 From: seth at icir.org (Seth Hall) Date: Wed, 4 Feb 2015 09:44:07 -0500 Subject: [Bro] Sending logs to remote public cloud entity In-Reply-To: <05419566-D78C-4A9D-A422-D5B76F2B5A02@gmail.com> References: <05419566-D78C-4A9D-A422-D5B76F2B5A02@gmail.com> Message-ID: <1E7D35F9-8F72-4727-A3A4-7B8257B2223F@icir.org> > On Feb 3, 2015, at 7:21 PM, shasubra1 at gmail.com wrote: > > I read some documentation about Broccoli whereby I can configure an SSL tunnel by furnishing the manager > with the a public cert, key and CA. I have not found much documentation nor > discussion on this kind of a setup usage. Broccoli is going to be marked as deprecated beginning with the next release so it?s on it?s last legs at the moment. There also isn?t a way with Broccoli to hook into the remote logging. Only Bro can send or receive logs remotely. Our replacement mechanism for Broccoli however will actually be able to send and receive logs remotely in non-Bro processes. > - is this the recommended approach to send logs to a remote public cloud entity There isn?t a recommended approach to this at the moment. I know of some companies using Bro and forwarding logs off to public cloud servers but they tend to compress and shuttle logs in bulk over other mechanisms (scp for example). I don?t know of anyone streaming logs off to cloud servers. > - the alternative is to send syslog?s but then I would need to set up stunnel or some other Yeah, that doesn?t sound fun. We also don?t support writing logs directly to syslog so you?d still end up running something else to pick the logs up off the disk and forward them off to the remote server. > - will the Bro manager scale to receive logs from multiple workers (like 10) > - I can work around this by running multiple Bro managers listening on different ports  Yes, this is fine. There are quite a few people around that have dozens of Bro processes sending logs to the manager and it takes them just fine. > - will the logs be written into the normal place on disk with the default writer Yep. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From Emmanuel.TORQUATO at monext.net Wed Feb 4 06:54:29 2015 From: Emmanuel.TORQUATO at monext.net (Emmanuel TORQUATO) Date: Wed, 4 Feb 2015 15:54:29 +0100 Subject: [Bro] Protocol decoder Message-ID: Hello, I would like to know what is the best approach for protocol decoder. Is it better to use BinPAC or BinPAC++ ? When does BinPAC++ will be supported for production? Thanks Emmanuel. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150204/123cef39/attachment.html From robin at icir.org Wed Feb 4 08:28:03 2015 From: robin at icir.org (Robin Sommer) Date: Wed, 4 Feb 2015 08:28:03 -0800 Subject: [Bro] Protocol decoder In-Reply-To: References: Message-ID: <20150204162803.GX35937@icir.org> On Wed, Feb 04, 2015 at 15:54 +0100, Emmanuel TORQUATO wrote: > I would like to know what is the best approach for protocol decoder. Is it better to use BinPAC or BinPAC++ ? I would say it depends on your use case. If it's indeed "production", then BinPAC. BinPAC++ is still in prototype state. If you're just experimenting with some protocol though, it can make sense to give it a try. > When does BinPAC++ will be supported for production? Unclear, there's signficant work left. It's at the state where the research work has concluded and produced a solid prototype, but taking it to production will require additional resources that we don't have right now unfortunately. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From td66bshwu at gmail.com Wed Feb 4 16:27:45 2015 From: td66bshwu at gmail.com (Kang) Date: Thu, 05 Feb 2015 10:57:45 +1030 Subject: [Bro] resp_bytes bug Message-ID: <54D2B901.8080101@gmail.com> Hello. I've been using Bro a lot lately and recently I've started noticing some weird connection sizes. For instance a single connection may have a resp_bytes of over 1000GB, far more than is possible given the circumstances. Three weirdness notifications seem to pop up along with this error, although not always all three at once. They are: SYN_seq_jump, SYN_inside_connection, & TCP_ack_underflow_or_misorder. I've managed to capture an instance of bug happening and have attached the dump to this email. If you run the dump through bro it should show a resp_bytes of almost 4GB for this connection, despite the capture only being a couple KB. Could you please help me understand what is happening her and perhaps fix the bug? Thank you -------------- next part -------------- A non-text attachment was scrubbed... Name: bad.connection.pcap Type: application/vnd.tcpdump.pcap Size: 1966 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150205/88038fad/attachment.bin From Emmanuel.TORQUATO at monext.net Thu Feb 5 03:06:58 2015 From: Emmanuel.TORQUATO at monext.net (Emmanuel TORQUATO) Date: Thu, 5 Feb 2015 12:06:58 +0100 Subject: [Bro] BinPAC and hexadecimal representation Message-ID: Hello everyone, I 'm working on a protocol decoder and would like to return an hexadecimal representation of a raw stream bytes. By reading some BinPac code I found the function "bytestring_to_val" to decode a stream as ASCII and I wonder if such a function exists for hexadecimal representation like will do ascii.hexlify() as example in python. Thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150205/13748e55/attachment.html From jsiwek at illinois.edu Thu Feb 5 07:16:21 2015 From: jsiwek at illinois.edu (Siwek, Jon) Date: Thu, 5 Feb 2015 15:16:21 +0000 Subject: [Bro] BinPAC and hexadecimal representation In-Reply-To: References: Message-ID: > On Feb 5, 2015, at 5:06 AM, Emmanuel TORQUATO wrote: > > By reading some BinPac code I found the function ?bytestring_to_val? to decode a stream as ASCII and I wonder if such a function exists for hexadecimal representation like will do ascii.hexlify() as example in python. Not sure exactly what you need, but the get_escaped_string() function in Bro?s src/util.cc may help or give you ideas about how to write what you need. - Jon From Emmanuel.TORQUATO at monext.net Thu Feb 5 08:09:23 2015 From: Emmanuel.TORQUATO at monext.net (Emmanuel TORQUATO) Date: Thu, 5 Feb 2015 17:09:23 +0100 Subject: [Bro] BinPAC and hexadecimal representation In-Reply-To: References: Message-ID: Thanks Jon, In fact I would like to have as result a string which is not a hexa convertion to ASCII put just a string that shows the data as an Hex representation For example, if in binPac we have parsed data to a bytestring object, I would like to just put the hex value to string representation not an ASCII convertion. For example a 4 bytes data like this 03 af 3c 4c parsed as bytestring in bin pac would be converted to a string like this "\0x03\0xaf\0x3c\0x4c" Regards, Emmanuel. -----Message d'origine----- De?: Siwek, Jon [mailto:jsiwek at illinois.edu] Envoy??: jeudi 5 f?vrier 2015 16:16 ??: Emmanuel TORQUATO Cc?: bro at bro.org Objet?: Re: [Bro] BinPAC and hexadecimal representation > On Feb 5, 2015, at 5:06 AM, Emmanuel TORQUATO wrote: > > By reading some BinPac code I found the function ?bytestring_to_val? to decode a stream as ASCII and I wonder if such a function exists for hexadecimal representation like will do ascii.hexlify() as example in python. Not sure exactly what you need, but the get_escaped_string() function in Bro?s src/util.cc may help or give you ideas about how to write what you need. - Jon From dj.root at netronome.com Thu Feb 5 11:30:31 2015 From: dj.root at netronome.com (DJ Root) Date: Thu, 5 Feb 2015 14:30:31 -0500 Subject: [Bro] Use PFRING_ZC for Bro In-Reply-To: References: <7D934230A65C8F498E98A3E6764DB983BE06E5C7@UCCS-EX3.uccs.edu> Message-ID: <103F38D8-C1D2-49D4-B05F-2CD31BB0FB14@netronome.com> Clement, We have Bro running on a 2 unit stack (4RU total) and see 17-18G of Bro steady state, peak 40-45G with few packet drops (~2%). To achieve this, we have 32 worker threads on each 2U appliance; of the remaining cores, 4 are for NIC management and the other 12 can be used for other applications. Bro is not modified at all. If you (or anyone else) would like to discuss further, please feel free to send me a private email. Regards, DJ Root > On Jan 27, 2015, at 4:42 PM, Clement Chen wrote: > > I was seeing 60% packet loss rate. After some aggressive BPF filtering, it went down to about 15%-20%. > > Are you using a big box? Mine is 24 core CPU with 64GB mem. There is an email thread about Bro with 10G card and many people also see pretty significant packet loss. > > It would be great if you can share your configs and also your traffic throughput. > > Thanks. > > -Clement > > On Tue, Jan 27, 2015 at 1:35 PM, Greg Williams > wrote: > Why do you want to use it?? I?m using security onion with Bro and 2x2 Intel x520 10Gb cards and have no packet loss with the base SO configuration. <> > > > Greg Williams, M.E., ISA, GPEN, GCFE > > Director of Networks and Infrastructure > Interim IT Security Manager/Information Security Officer/HIPAA Security Officer > University of Colorado Colorado Springs - Department of Information Technology > Phone: 719-255-3211 > > > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org ] On Behalf Of Clement Chen > Sent: Tuesday, January 27, 2015 2:22 PM > To: bro at bro.org > Subject: [Bro] Use PFRING_ZC for Bro > > > > Hi all, > > > > I am trying to use PFRING_ZC for Bro in my security onion box. I got the license from ntop but there was little document on how to enable this. > > > > Would appreciate any help/pointer to docs. I will compile a step-by-step instructions if I get this working. > > > > I have the Intel 82599EB 10G card and the ixgbe-zc driver installed. > > > > #dkms status > > ixgbe-zc, 3.22.3, 3.13.0-44-generic, x86_64: installed > > pf_ring, 6, 3.13.0-35-generic, x86_64: installed > > pf_ring, 6, 3.13.0-44-generic, x86_64: installed (WARNING! Diff between built and installed module!) > > pfring, 6.0.3, 3.13.0-44-generic, x86_64: installed > > > > not sure what to do next and how to enable it for Bro. > > > > Thanks. > > > > -Clement > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150205/b3395719/attachment-0001.html From jdonnelly at dyn.com Fri Feb 6 05:09:36 2015 From: jdonnelly at dyn.com (John Donnelly) Date: Fri, 6 Feb 2015 07:09:36 -0600 Subject: [Bro] New plugin usage: Error: " value used but not set" Message-ID: Hi I have a SW module that worked using the older plugin module from 2.2.135 Bro and I recently merged it to Bro 2.3.397 and when I run my Bro script that uses it I get: /usr/local/bro/share/bro/base/protocols/dns/telemetry_speed.bro, line 454: value used but not set (dns_telemetry_set_options) dns_telemetry_set_options is a function contained in my module in a events.bif file. The file is compiled. Any help in diagnosing this err would be appreciated . -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150206/d1694659/attachment.html From jdonnelly at dyn.com Fri Feb 6 05:19:31 2015 From: jdonnelly at dyn.com (John Donnelly) Date: Fri, 6 Feb 2015 07:19:31 -0600 Subject: [Bro] New plugin usage: Error: " value used but not set" In-Reply-To: References: Message-ID: I see the library plugin being created : make[3]: Entering directory `/work/jpd/dyn/src/bro-fork2/bro/build' [ 27%] Building CXX object src/analyzer/protocol/dns_telemetry/CMakeFiles/plugin-Bro-DNS_TELEMETRY.dir/DNS.cc.o [ 27%] Building CXX object src/analyzer/protocol/dns_telemetry/CMakeFiles/plugin-Bro-DNS_TELEMETRY.dir/events.bif.cc.o [ 27%] Building CXX object src/analyzer/protocol/dns_telemetry/CMakeFiles/plugin-Bro-DNS_TELEMETRY.dir/events.bif.init.cc.o Linking CXX static library libplugin-Bro-DNS_TELEMETRY.a On Fri, Feb 6, 2015 at 7:09 AM, John Donnelly wrote: > Hi > > I have a SW module that worked using the older plugin module from 2.2.135 > Bro and I recently merged it to Bro 2.3.397 and when I run my Bro script > that uses it I get: > > /usr/local/bro/share/bro/base/protocols/dns/telemetry_speed.bro, line 454: > value used but not set (dns_telemetry_set_options) > > dns_telemetry_set_options is a function contained in my module in > a events.bif file. The file is compiled. > > Any help in diagnosing this err would be appreciated . > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150206/11385a0c/attachment.html From seth at icir.org Fri Feb 6 05:29:14 2015 From: seth at icir.org (Seth Hall) Date: Fri, 6 Feb 2015 08:29:14 -0500 Subject: [Bro] New plugin usage: Error: " value used but not set" In-Reply-To: References: Message-ID: If your plugin in the BRO_PLUGIN_PATH? Also, if your Bro script is something that you would always want to load with your module, you can include it with the module and make it automatically load when Bro starts up. This lets you completely keep any evidence of your plugin out of Bro?s normal installation (I see you added your script to the base/ directory which you probably don?t want to do). .Seth > On Feb 6, 2015, at 8:19 AM, John Donnelly wrote: > > I see the library plugin being created : > > make[3]: Entering directory `/work/jpd/dyn/src/bro-fork2/bro/build' > [ 27%] Building CXX object src/analyzer/protocol/dns_telemetry/CMakeFiles/plugin-Bro-DNS_TELEMETRY.dir/DNS.cc.o > [ 27%] Building CXX object src/analyzer/protocol/dns_telemetry/CMakeFiles/plugin-Bro-DNS_TELEMETRY.dir/events.bif.cc.o > [ 27%] Building CXX object src/analyzer/protocol/dns_telemetry/CMakeFiles/plugin-Bro-DNS_TELEMETRY.dir/events.bif.init.cc.o > Linking CXX static library libplugin-Bro-DNS_TELEMETRY.a > > > On Fri, Feb 6, 2015 at 7:09 AM, John Donnelly wrote: > Hi > > I have a SW module that worked using the older plugin module from 2.2.135 Bro and I recently merged it to Bro 2.3.397 and when I run my Bro script that uses it I get: > > /usr/local/bro/share/bro/base/protocols/dns/telemetry_speed.bro, line 454: value used but not set (dns_telemetry_set_options) > > dns_telemetry_set_options is a function contained in my module in a events.bif file. The file is compiled. > > Any help in diagnosing this err would be appreciated . > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jdonnelly at dyn.com Fri Feb 6 05:41:07 2015 From: jdonnelly at dyn.com (John Donnelly) Date: Fri, 6 Feb 2015 07:41:07 -0600 Subject: [Bro] New plugin usage: Error: " value used but not set" In-Reply-To: References: Message-ID: That directory is empty: bro -h : $BRO_PLUGIN_PATH | plugin search path (/usr/local/bro/lib/bro/plugins) ls /usr/local/bro/lib/bro/plugins ls: cannot access /usr/local/bro/lib/bro/plugins: No such file or directory On Fri, Feb 6, 2015 at 7:29 AM, Seth Hall wrote: > If your plugin in the BRO_PLUGIN_PATH? > > Also, if your Bro script is something that you would always want to load > with your module, you can include it with the module and make it > automatically load when Bro starts up. This lets you completely keep any > evidence of your plugin out of Bro?s normal installation (I see you added > your script to the base/ directory which you probably don?t want to do). > > .Seth > > > > On Feb 6, 2015, at 8:19 AM, John Donnelly wrote: > > > > I see the library plugin being created : > > > > make[3]: Entering directory `/work/jpd/dyn/src/bro-fork2/bro/build' > > [ 27%] Building CXX object > src/analyzer/protocol/dns_telemetry/CMakeFiles/plugin-Bro-DNS_TELEMETRY.dir/DNS.cc.o > > [ 27%] Building CXX object > src/analyzer/protocol/dns_telemetry/CMakeFiles/plugin-Bro-DNS_TELEMETRY.dir/events.bif.cc.o > > [ 27%] Building CXX object > src/analyzer/protocol/dns_telemetry/CMakeFiles/plugin-Bro-DNS_TELEMETRY.dir/events.bif.init.cc.o > > Linking CXX static library libplugin-Bro-DNS_TELEMETRY.a > > > > > > On Fri, Feb 6, 2015 at 7:09 AM, John Donnelly wrote: > > Hi > > > > I have a SW module that worked using the older plugin module from > 2.2.135 Bro and I recently merged it to Bro 2.3.397 and when I run my Bro > script that uses it I get: > > > > /usr/local/bro/share/bro/base/protocols/dns/telemetry_speed.bro, line > 454: value used but not set (dns_telemetry_set_options) > > > > dns_telemetry_set_options is a function contained in my module in a > events.bif file. The file is compiled. > > > > Any help in diagnosing this err would be appreciated . > > > > > > > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150206/1c8c7f0d/attachment.html From jdonnelly at dyn.com Fri Feb 6 05:48:29 2015 From: jdonnelly at dyn.com (John Donnelly) Date: Fri, 6 Feb 2015 07:48:29 -0600 Subject: [Bro] New plugin usage: Error: " value used but not set" In-Reply-To: References: Message-ID: bro -N | grep DNS opened 'debug.log' debugging output Bro::DNS - DNS analyzer (built-in) Bro::DNS_Telemetry - DNS analyzer Telemetry (built-in) On Fri, Feb 6, 2015 at 7:41 AM, John Donnelly wrote: > That directory is empty: > > bro -h : > > $BRO_PLUGIN_PATH | plugin search path > (/usr/local/bro/lib/bro/plugins) > > ls /usr/local/bro/lib/bro/plugins > ls: cannot access /usr/local/bro/lib/bro/plugins: No such file or directory > > > > > > On Fri, Feb 6, 2015 at 7:29 AM, Seth Hall wrote: > >> If your plugin in the BRO_PLUGIN_PATH? >> >> Also, if your Bro script is something that you would always want to load >> with your module, you can include it with the module and make it >> automatically load when Bro starts up. This lets you completely keep any >> evidence of your plugin out of Bro?s normal installation (I see you added >> your script to the base/ directory which you probably don?t want to do). >> >> .Seth >> >> >> > On Feb 6, 2015, at 8:19 AM, John Donnelly wrote: >> > >> > I see the library plugin being created : >> > >> > make[3]: Entering directory `/work/jpd/dyn/src/bro-fork2/bro/build' >> > [ 27%] Building CXX object >> src/analyzer/protocol/dns_telemetry/CMakeFiles/plugin-Bro-DNS_TELEMETRY.dir/DNS.cc.o >> > [ 27%] Building CXX object >> src/analyzer/protocol/dns_telemetry/CMakeFiles/plugin-Bro-DNS_TELEMETRY.dir/events.bif.cc.o >> > [ 27%] Building CXX object >> src/analyzer/protocol/dns_telemetry/CMakeFiles/plugin-Bro-DNS_TELEMETRY.dir/events.bif.init.cc.o >> > Linking CXX static library libplugin-Bro-DNS_TELEMETRY.a >> > >> > >> > On Fri, Feb 6, 2015 at 7:09 AM, John Donnelly >> wrote: >> > Hi >> > >> > I have a SW module that worked using the older plugin module from >> 2.2.135 Bro and I recently merged it to Bro 2.3.397 and when I run my Bro >> script that uses it I get: >> > >> > /usr/local/bro/share/bro/base/protocols/dns/telemetry_speed.bro, line >> 454: value used but not set (dns_telemetry_set_options) >> > >> > dns_telemetry_set_options is a function contained in my module in a >> events.bif file. The file is compiled. >> > >> > Any help in diagnosing this err would be appreciated . >> > >> > >> > >> > >> > _______________________________________________ >> > Bro mailing list >> > bro at bro-ids.org >> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> -- >> Seth Hall >> International Computer Science Institute >> (Bro) because everyone has a network >> http://www.bro.org/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150206/e7475da4/attachment-0001.html From seth at icir.org Fri Feb 6 06:18:12 2015 From: seth at icir.org (Seth Hall) Date: Fri, 6 Feb 2015 09:18:12 -0500 Subject: [Bro] New plugin usage: Error: " value used but not set" In-Reply-To: References: Message-ID: <71E19C8C-B968-4AE9-808E-2EA5DCF6AD5E@icir.org> > On Feb 6, 2015, at 8:48 AM, John Donnelly wrote: > > bro -N | grep DNS > opened 'debug.log' debugging output > Bro::DNS - DNS analyzer (built-in) > Bro::DNS_Telemetry - DNS analyzer Telemetry (built-in) Oh, I?m sorry. I thought you were writing this as an external/dynamic plugin. Have you tried looking at the output if you load the misc/loaded-scripts script? That should show a file named something like this getting loaded? scripts/base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jdonnelly at dyn.com Fri Feb 6 06:30:44 2015 From: jdonnelly at dyn.com (John Donnelly) Date: Fri, 6 Feb 2015 08:30:44 -0600 Subject: [Bro] New plugin usage: Error: " value used but not set" In-Reply-To: <71E19C8C-B968-4AE9-808E-2EA5DCF6AD5E@icir.org> References: <71E19C8C-B968-4AE9-808E-2EA5DCF6AD5E@icir.org> Message-ID: No indication it gets loaded: find /usr/local/bro | grep TELE /usr/local/bro/share/bro/base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro root at dyn-x64-01:/work/jpd/dyn/src/bro-fork2/bro# export BRO_PLUGIN_PATH=/usr/local/bro/share/bro/base/bif/plugins /usr/local/bro/bin/bro -i lo -i eth0 -i eth1 -b -C /usr/local/bro/share/bro/base/protocols/dns/telemetry_speed.bro listening on lo, capture length 8192 bytes listening on eth0, capture length 8192 bytes listening on eth1, capture length 8192 bytes 1423232941.050872 error in /usr/local/bro/share/bro/base/protocols/dns/telemetry_speed.bro, line 454: value used but not set (dns_telemetry_set_options) 1423232941.050872 error in /usr/local/bro/share/bro/base/protocols/dns/telemetry_speed.bro, line 508: value used but not set (dns_telemetry_load_anchor_map) 1423232941.050872 error in /usr/local/bro/share/bro/base/protocols/dns/telemetry_speed.bro, line 524: value used but not set (dns_telemetry_get_metrics) On Fri, Feb 6, 2015 at 8:18 AM, Seth Hall wrote: > > > On Feb 6, 2015, at 8:48 AM, John Donnelly wrote: > > > > bro -N | grep DNS > > opened 'debug.log' debugging output > > Bro::DNS - DNS analyzer (built-in) > > Bro::DNS_Telemetry - DNS analyzer Telemetry (built-in) > > Oh, I?m sorry. I thought you were writing this as an external/dynamic > plugin. > > Have you tried looking at the output if you load the misc/loaded-scripts > script? That should show a file named something like this getting loaded? > > scripts/base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro > > .Seth > > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150206/057763d3/attachment.html From jdonnelly at dyn.com Fri Feb 6 07:36:16 2015 From: jdonnelly at dyn.com (John Donnelly) Date: Fri, 6 Feb 2015 09:36:16 -0600 Subject: [Bro] New plugin usage: Error: " value used but not set" In-Reply-To: References: <71E19C8C-B968-4AE9-808E-2EA5DCF6AD5E@icir.org> Message-ID: using -B plugins I see: Registering component DNS_TELEMETRY (tag 13/0) On Fri, Feb 6, 2015 at 8:30 AM, John Donnelly wrote: > No indication it gets loaded: > > find /usr/local/bro | grep TELE > /usr/local/bro/share/bro/base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro > root at dyn-x64-01:/work/jpd/dyn/src/bro-fork2/bro# export > BRO_PLUGIN_PATH=/usr/local/bro/share/bro/base/bif/plugins > > /usr/local/bro/bin/bro -i lo -i eth0 -i eth1 -b -C > /usr/local/bro/share/bro/base/protocols/dns/telemetry_speed.bro > > listening on lo, capture length 8192 bytes > > listening on eth0, capture length 8192 bytes > > listening on eth1, capture length 8192 bytes > > 1423232941.050872 error in > /usr/local/bro/share/bro/base/protocols/dns/telemetry_speed.bro, line 454: > value used but not set (dns_telemetry_set_options) > 1423232941.050872 error in > /usr/local/bro/share/bro/base/protocols/dns/telemetry_speed.bro, line 508: > value used but not set (dns_telemetry_load_anchor_map) > 1423232941.050872 error in > /usr/local/bro/share/bro/base/protocols/dns/telemetry_speed.bro, line 524: > value used but not set (dns_telemetry_get_metrics) > > > > > > On Fri, Feb 6, 2015 at 8:18 AM, Seth Hall wrote: > >> >> > On Feb 6, 2015, at 8:48 AM, John Donnelly wrote: >> > >> > bro -N | grep DNS >> > opened 'debug.log' debugging output >> > Bro::DNS - DNS analyzer (built-in) >> > Bro::DNS_Telemetry - DNS analyzer Telemetry (built-in) >> >> Oh, I?m sorry. I thought you were writing this as an external/dynamic >> plugin. >> >> Have you tried looking at the output if you load the misc/loaded-scripts >> script? That should show a file named something like this getting loaded? >> >> scripts/base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro >> >> .Seth >> >> >> -- >> Seth Hall >> International Computer Science Institute >> (Bro) because everyone has a network >> http://www.bro.org/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150206/d2716b33/attachment.html From robin at icir.org Fri Feb 6 07:49:30 2015 From: robin at icir.org (Robin Sommer) Date: Fri, 6 Feb 2015 07:49:30 -0800 Subject: [Bro] New plugin usage: Error: " value used but not set" In-Reply-To: References: Message-ID: <20150206154930.GB73984@icir.org> On Fri, Feb 06, 2015 at 07:48 -0600, John Donnelly wrote: > bro -N | grep DNS Does -NN show your function? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jdonnelly at dyn.com Fri Feb 6 08:00:14 2015 From: jdonnelly at dyn.com (John Donnelly) Date: Fri, 6 Feb 2015 10:00:14 -0600 Subject: [Bro] New plugin usage: Error: " value used but not set" In-Reply-To: <20150206154930.GB73984@icir.org> References: <20150206154930.GB73984@icir.org> Message-ID: No . -NN only shows the Plugin Name: Bro::DNS_Telemetry - DNS analyzer Telemetry (built-in) [Analyzer] Contents_DNS_Telemetry (enabled) [Analyzer] DNS_TELEMETRY (ANALYZER_DNS_TELEMETRY, enabled) I turned on all -B options and gathered this from debug.log and what -NN shows Made IdentifierInfo dns_telemetry_set_options, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Filter id 'dns_telemetry_set_options' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function On Fri, Feb 6, 2015 at 9:49 AM, Robin Sommer wrote: > On Fri, Feb 06, 2015 at 07:48 -0600, John Donnelly wrote: > > > bro -N | grep DNS > > Does -NN show your function? > > Robin > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150206/1d41b222/attachment-0001.html -------------- next part -------------- Registering component DNS_TELEMETRY (tag 13/0) Made internal IdentifierInfo Analyzer::ANALYZER_DNS_TELEMETRY Made internal IdentifierInfo Analyzer::ANALYZER_CONTENTS_DNS_TELEMETRY Made ScriptInfo base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Added script dependency base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro for base/bif/plugins/__load__.bro Made IdentifierInfo dns_telemetry_message, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_request, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_rejected, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_query_reply, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_EDNS_addl, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_TSIG_addl, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_end, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo non_dns_telemetry_request, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_count, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_totals, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_anyrd_info, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_client_info, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_zone_info, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_owner_info, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_qname_info, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_detail_info, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_zone_info_list, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_fire_counts, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_fire_totals, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_fire_anyrd, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_fire_clients, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_fire_zones, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_fire_owners, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_fire_qnames, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_fire_details, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_get_do_counts, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_get_do_totals, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_get_do_details, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_get_do_anyrd, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_get_do_clients, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_get_do_zones, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_get_do_qnames, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_set_do_counts, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_get_sample_rate, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_set_sample_rate, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_set_do_details, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_set_do_details_statsd, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_set_do_details_redis, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_set_do_totals, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_set_do_zones, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_load_anchor_map, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_set_do_owners, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_set_do_qnames, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_set_do_anyrd, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_set_do_clients, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_set_do_log_all, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_get_do_log_all, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_set_node_id, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_set_options, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Made IdentifierInfo dns_telemetry_get_metrics, in script base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Added module usage GLOBAL in base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro Filter id 'dns_telemetry_EDNS_addl' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a event Filter id 'dns_telemetry_TSIG_addl' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a event Filter id 'dns_telemetry_anyrd_info' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a event Filter id 'dns_telemetry_client_info' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a event Filter id 'dns_telemetry_count' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a event Filter id 'dns_telemetry_detail_info' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a event Filter id 'dns_telemetry_end' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a event Filter id 'dns_telemetry_fire_anyrd' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_fire_clients' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_fire_counts' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_fire_details' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_fire_owners' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_fire_qnames' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_fire_totals' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_fire_zones' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_get_do_anyrd' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_get_do_clients' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_get_do_counts' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_get_do_details' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_get_do_log_all' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_get_do_qnames' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_get_do_totals' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_get_do_zones' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_get_metrics' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_get_sample_rate' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_load_anchor_map' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_message' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a event Filter id 'dns_telemetry_owner_info' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a event Filter id 'dns_telemetry_qname_info' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a event Filter id 'dns_telemetry_query_reply' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a event Filter id 'dns_telemetry_rejected' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a event Filter id 'dns_telemetry_request' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a event Filter id 'dns_telemetry_set_do_anyrd' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_set_do_clients' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_set_do_counts' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_set_do_details' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_set_do_details_redis' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_set_do_details_statsd' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_set_do_log_all' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_set_do_owners' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_set_do_qnames' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_set_do_totals' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_set_do_zones' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_set_node_id' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_set_options' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_set_sample_rate' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'dns_telemetry_totals' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a event Filter id 'dns_telemetry_zone_info' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a event Filter id 'dns_telemetry_zone_info_list' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function Filter id 'non_dns_telemetry_request' in 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a event DNS_TELEMETRY (enabled) Registering analyzer DNS_TELEMETRY for port 53/2 operation: Analyzer::ports[Analyzer::ANALYZER_DNS_TELEMETRY] = {} [val] (#dyn-x64-01#29199#69) [] Registering analyzer DNS_TELEMETRY for port 53/1 UDP[16] added child DNS_TELEMETRY[18] 192.168.1.111:55406 > 192.168.1.123:53 activated DNS_TELEMETRY analyzer due to port 53 UDP[20] added child DNS_TELEMETRY[22] UDP[24] added child DNS_TELEMETRY[26] UDP[28] added child DNS_TELEMETRY[30] bro -NN Bro::DNS_Telemetry - DNS analyzer Telemetry (built-in) [Analyzer] Contents_DNS_Telemetry (enabled) [Analyzer] DNS_TELEMETRY (ANALYZER_DNS_TELEMETRY, enabled) From jdonnelly at dyn.com Fri Feb 6 08:17:33 2015 From: jdonnelly at dyn.com (John Donnelly) Date: Fri, 6 Feb 2015 10:17:33 -0600 Subject: [Bro] New plugin usage: Error: " value used but not set" In-Reply-To: References: <20150206154930.GB73984@icir.org> Message-ID: My Pluggin - I am using the same InstantiateAnalyzer method as I did in 2.2.135: class Plugin : public plugin::Plugin { public: plugin::Configuration Configure() { AddComponent(new ::analyzer::Component("DNS_TELEMETRY", ::analyzer::dns_telemetry::DNS_Telemetry_Analyzer::InstantiateAnalyzer)); AddComponent(new ::analyzer::Component("Contents_DNS_Telemetry", 0)); plugin::Configuration config; config.name = "Bro::DNS_Telemetry"; config.description = "DNS analyzer Telemetry"; return config; } On Fri, Feb 6, 2015 at 10:00 AM, John Donnelly wrote: > No . -NN only shows the Plugin Name: > > > > Bro::DNS_Telemetry - DNS analyzer Telemetry (built-in) > [Analyzer] Contents_DNS_Telemetry (enabled) > [Analyzer] DNS_TELEMETRY (ANALYZER_DNS_TELEMETRY, enabled) > > > I turned on all -B options and gathered this from debug.log and what -NN > shows > > > Made IdentifierInfo dns_telemetry_set_options, in script > base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro > > Filter id 'dns_telemetry_set_options' in > 'base/bif/plugins/Bro_DNS_TELEMETRY.events.bif.bro' as a function > > > > > On Fri, Feb 6, 2015 at 9:49 AM, Robin Sommer wrote: > >> On Fri, Feb 06, 2015 at 07:48 -0600, John Donnelly wrote: >> >> > bro -N | grep DNS >> >> Does -NN show your function? >> >> Robin >> >> -- >> Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150206/6dfd0b19/attachment.html From td66bshwu at gmail.com Fri Feb 6 18:44:52 2015 From: td66bshwu at gmail.com (Lachlan Kang) Date: Sat, 7 Feb 2015 13:14:52 +1030 Subject: [Bro] Bro BitTorrent Message-ID: Are there any extensions to bro or special options that allow you to detect BitTorrent? >From what I've seen it's never detected, instead classified as having an unknown service. From seth at icir.org Fri Feb 6 20:08:58 2015 From: seth at icir.org (Seth Hall) Date: Fri, 6 Feb 2015 23:08:58 -0500 Subject: [Bro] Bro BitTorrent In-Reply-To: References: Message-ID: <329142C0-32BA-4C54-819F-06BF1FBEED7D@icir.org> > On Feb 6, 2015, at 9:44 PM, Lachlan Kang wrote: > > Are there any extensions to bro or special options that allow you to > detect BitTorrent? There is a bittorrent analyzer but it has a few issues and there aren?t base Bro scripts for it right now. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From takedownz at gmail.com Sat Feb 7 03:55:22 2015 From: takedownz at gmail.com (takedown) Date: Sat, 7 Feb 2015 14:55:22 +0300 Subject: [Bro] (no subject) Message-ID: Hello everyone, Today a lot of indicators posted in form of reports and i got tired to convert this information manually into Bro Intel files, so i decided to write a little tool to automate this process. Introducing Bro intel generator - script to generate bro intel files from pdf or html reports. Check it out at https://github.com/exp0se/bro-intel-generator and tell me what you think. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150207/c0a0333c/attachment.html From abhall1 at yahoo.com Sun Feb 8 16:53:11 2015 From: abhall1 at yahoo.com (Adam Hall) Date: Mon, 9 Feb 2015 00:53:11 +0000 (UTC) Subject: [Bro] BPF Filter Help Message-ID: <795783142.1660128.1423443191393.JavaMail.yahoo@mail.yahoo.com> Good Evening Bro Team, I have ran into an issue with using the BPF packet filter. ?I have had the same issue using Bro2.2, 2.3.1, and 2.3.411 on both Ubuntu 14.04 and Gentoo 3.0.2. ?The way I am calling the packet filter is through the local.bro file using this command: # Packet Filter optionsevent bro_init()? ? ? ? {? ? ? ? PacketFilter::exclude("ignore_this_conn","host 10.8.0.85 and port 53");? ? ? ? } ?and you can see it accepted the filter using "broctl diag": 1423442280.253256 ? ? ? bro ? ? (ip or not ip) and (not (host 10.8.0.85 and port 53)) ? T ? ? ? T?If you used an incorrect bpf filter like "source.host 10.8.0.85" the "broctl diag" would give you nothing: 1423442280.253847 ? ? ? bro ? ? (ip or not ip) ? T ? ? ? T What I am currently trying to do is exclude dns traffic with a destination of this host and port 53: (dst host 10.8.0.85 and dst port 53) When I add this in the exclude statement the bpf is accepted 1423442632.139980 ? ? ? bro ? ? (ip or not ip) and (not (dst host 10.8.0.85 and dst port 53)) ? T ? ? ? T However, the traffic is still being allowed and not excluded 1423442692.141824 ? ? ? C7pSulFJiU150KhFk ? ? ? 10.8.1.43 ? ? ? 46088 ? 10.8.0.85 ? ? ? 53 ? ? ?udp ? ? 33647 ? - ? ? ? - ? ? ? - ? ? ? - ? ? - The only way I have been able to successfully get this to work is by defining only "host" or "port", I have not been able to get this to be successful using a "src host", "dst host", "src port", or "dst port". This creates a problem to the point it's almost unusable to me as I cannot ignore all traffic for "host 10.8.0.85 and port 53". Any help with this would be greatly appreciated! Thank You, Adam B. Hall | CCNA Senior Security Analyst Office:?1-800-538-9357?x 122 Mobile:?1-904-303-3198 Quadrant Information Security 4651 Salisbury Road, Suite 185 | Jacksonville, FL 32256 See our Quadrant Video https://quadrantsec.com/SaganMSSP/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150209/89fd39b1/attachment.html From seth at icir.org Mon Feb 9 06:19:11 2015 From: seth at icir.org (Seth Hall) Date: Mon, 9 Feb 2015 09:19:11 -0500 Subject: [Bro] BPF Filter Help In-Reply-To: <795783142.1660128.1423443191393.JavaMail.yahoo@mail.yahoo.com> References: <795783142.1660128.1423443191393.JavaMail.yahoo@mail.yahoo.com> Message-ID: <45CA62D7-4066-4611-B946-E13C4751E197@icir.org> > On Feb 8, 2015, at 7:53 PM, Adam Hall wrote: > > 1423442632.139980 bro (ip or not ip) and (not (dst host 10.8.0.85 and dst port 53)) T T > The only way I have been able to successfully get this to work is by defining only "host" or "port", I have not been able to get this to be successful using a "src host", "dst host", "src port", or "dst port". > > This creates a problem to the point it's almost unusable to me as I cannot ignore all traffic for "host 10.8.0.85 and port 53". It?s unlikely that you are ever going to want to use the ?src? or ?dst? modifiers in filters meant for Bro. Bro is intended to look at both directions of traffic to successfully analyze connections and you are intrinsically breaking that when you use src or dst. What is the end result you?re trying to get to? You just don?t want to see dns traffic involving host 10.8.0.85? The problem here is that there is a disjoint in the semantics of BPF and the typical semantics of Bro. Bro operates on connections (and flows to a slightly lesser degree currently). BPF is completely oriented around packets. It causes these little confusions unfortunately. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From dj.root at netronome.com Mon Feb 9 08:49:24 2015 From: dj.root at netronome.com (DJ Root) Date: Mon, 9 Feb 2015 11:49:24 -0500 Subject: [Bro] BPF Filter Help In-Reply-To: <45CA62D7-4066-4611-B946-E13C4751E197@icir.org> References: <795783142.1660128.1423443191393.JavaMail.yahoo@mail.yahoo.com> <45CA62D7-4066-4611-B946-E13C4751E197@icir.org> Message-ID: Adam, What you describe here is a fairly straightforward match/action rule - in your case you want perform some action when you receive dst host 10.8.0.56 / dst port 53 / src host X / src port Y traffic - right? This operation can be done on an intelligent NIC card without any interaction with Bro (or BPF for that matter). If you would like more information on how we could help solve this problem, please email me privately. Regards, DJ Root > On Feb 9, 2015, at 9:19 AM, Seth Hall wrote: > > >> On Feb 8, 2015, at 7:53 PM, Adam Hall wrote: >> >> 1423442632.139980 bro (ip or not ip) and (not (dst host 10.8.0.85 and dst port 53)) T T >> The only way I have been able to successfully get this to work is by defining only "host" or "port", I have not been able to get this to be successful using a "src host", "dst host", "src port", or "dst port". >> >> This creates a problem to the point it's almost unusable to me as I cannot ignore all traffic for "host 10.8.0.85 and port 53". > > It?s unlikely that you are ever going to want to use the ?src? or ?dst? modifiers in filters meant for Bro. Bro is intended to look at both directions of traffic to successfully analyze connections and you are intrinsically breaking that when you use src or dst. > > What is the end result you?re trying to get to? You just don?t want to see dns traffic involving host 10.8.0.85? > > The problem here is that there is a disjoint in the semantics of BPF and the typical semantics of Bro. Bro operates on connections (and flows to a slightly lesser degree currently). BPF is completely oriented around packets. It causes these little confusions unfortunately. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From michalpurzynski1 at gmail.com Mon Feb 9 08:57:11 2015 From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=) Date: Mon, 9 Feb 2015 17:57:11 +0100 Subject: [Bro] BPF Filter Help In-Reply-To: References: <795783142.1660128.1423443191393.JavaMail.yahoo@mail.yahoo.com> <45CA62D7-4066-4611-B946-E13C4751E197@icir.org> Message-ID: Is there any reason why you cannot share this kind of information on the list, so everyone can benefit? Looks like man ethtool, right? On Mon, Feb 9, 2015 at 5:49 PM, DJ Root wrote: > Adam, > > What you describe here is a fairly straightforward match/action rule - in your case you want perform some action when you receive dst host 10.8.0.56 / dst port 53 / src host X / src port Y traffic - right? This operation can be done on an intelligent NIC card without any interaction with Bro (or BPF for that matter). > > If you would like more information on how we could help solve this problem, please email me privately. > > Regards, > DJ Root >> On Feb 9, 2015, at 9:19 AM, Seth Hall wrote: >> >> >>> On Feb 8, 2015, at 7:53 PM, Adam Hall wrote: >>> >>> 1423442632.139980 bro (ip or not ip) and (not (dst host 10.8.0.85 and dst port 53)) T T >>> The only way I have been able to successfully get this to work is by defining only "host" or "port", I have not been able to get this to be successful using a "src host", "dst host", "src port", or "dst port". >>> >>> This creates a problem to the point it's almost unusable to me as I cannot ignore all traffic for "host 10.8.0.85 and port 53". >> >> It?s unlikely that you are ever going to want to use the ?src? or ?dst? modifiers in filters meant for Bro. Bro is intended to look at both directions of traffic to successfully analyze connections and you are intrinsically breaking that when you use src or dst. >> >> What is the end result you?re trying to get to? You just don?t want to see dns traffic involving host 10.8.0.85? >> >> The problem here is that there is a disjoint in the semantics of BPF and the typical semantics of Bro. Bro operates on connections (and flows to a slightly lesser degree currently). BPF is completely oriented around packets. It causes these little confusions unfortunately. >> >> .Seth >> >> -- >> Seth Hall >> International Computer Science Institute >> (Bro) because everyone has a network >> http://www.bro.org/ >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From dj.root at netronome.com Mon Feb 9 10:20:36 2015 From: dj.root at netronome.com (DJ Root) Date: Mon, 9 Feb 2015 13:20:36 -0500 Subject: [Bro] BPF Filter Help In-Reply-To: References: <795783142.1660128.1423443191393.JavaMail.yahoo@mail.yahoo.com> <45CA62D7-4066-4611-B946-E13C4751E197@icir.org> Message-ID: Michal, No, there is not. However, I come from the vendor side and, therefore, don?t want to disrupt the integrity of a technology mailing list. That said we have Bro running in our lab, so our claims can be supported by real data and demos. As far as the question below is concerned, we (Netronome) have intelligent NICs and software to do match / action operations in hardware. We can set-up 5-tuple filtering in hardware which can address Adam?s problem. Result - BPF is off-loaded from the CPU; Bro is not changed, but now has more CPU cycles to process flows. Regards, DJ Root DJ Root Director of Sales, Americas East and EMEA Netronome, Inc. (617)686-0253 > On Feb 9, 2015, at 11:57 AM, Micha? Purzy?ski wrote: > > Is there any reason why you cannot share this kind of information on > the list, so everyone can benefit? > > Looks like man ethtool, right? > > On Mon, Feb 9, 2015 at 5:49 PM, DJ Root wrote: >> Adam, >> >> What you describe here is a fairly straightforward match/action rule - in your case you want perform some action when you receive dst host 10.8.0.56 / dst port 53 / src host X / src port Y traffic - right? This operation can be done on an intelligent NIC card without any interaction with Bro (or BPF for that matter). >> >> If you would like more information on how we could help solve this problem, please email me privately. >> >> Regards, >> DJ Root >>> On Feb 9, 2015, at 9:19 AM, Seth Hall wrote: >>> >>> >>>> On Feb 8, 2015, at 7:53 PM, Adam Hall wrote: >>>> >>>> 1423442632.139980 bro (ip or not ip) and (not (dst host 10.8.0.85 and dst port 53)) T T >>>> The only way I have been able to successfully get this to work is by defining only "host" or "port", I have not been able to get this to be successful using a "src host", "dst host", "src port", or "dst port". >>>> >>>> This creates a problem to the point it's almost unusable to me as I cannot ignore all traffic for "host 10.8.0.85 and port 53". >>> >>> It?s unlikely that you are ever going to want to use the ?src? or ?dst? modifiers in filters meant for Bro. Bro is intended to look at both directions of traffic to successfully analyze connections and you are intrinsically breaking that when you use src or dst. >>> >>> What is the end result you?re trying to get to? You just don?t want to see dns traffic involving host 10.8.0.85? >>> >>> The problem here is that there is a disjoint in the semantics of BPF and the typical semantics of Bro. Bro operates on connections (and flows to a slightly lesser degree currently). BPF is completely oriented around packets. It causes these little confusions unfortunately. >>> >>> .Seth >>> >>> -- >>> Seth Hall >>> International Computer Science Institute >>> (Bro) because everyone has a network >>> http://www.bro.org/ >>> >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From michalpurzynski1 at gmail.com Mon Feb 9 10:28:25 2015 From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=) Date: Mon, 9 Feb 2015 19:28:25 +0100 Subject: [Bro] BPF Filter Help In-Reply-To: References: <795783142.1660128.1423443191393.JavaMail.yahoo@mail.yahoo.com> <45CA62D7-4066-4611-B946-E13C4751E197@icir.org> Message-ID: Do you use X520 chip by accident? On Mon, Feb 9, 2015 at 7:20 PM, DJ Root wrote: > Michal, > > No, there is not. However, I come from the vendor side and, therefore, don?t want to disrupt the integrity of a technology mailing list. That said we have Bro running in our lab, so our claims can be supported by real data and demos. > > As far as the question below is concerned, we (Netronome) have intelligent NICs and software to do match / action operations in hardware. We can set-up 5-tuple filtering in hardware which can address Adam?s problem. Result - BPF is off-loaded from the CPU; Bro is not changed, but now has more CPU cycles to process flows. > > Regards, > DJ Root > > DJ Root > Director of Sales, Americas East and EMEA > Netronome, Inc. > (617)686-0253 > > > > >> On Feb 9, 2015, at 11:57 AM, Micha? Purzy?ski wrote: >> >> Is there any reason why you cannot share this kind of information on >> the list, so everyone can benefit? >> >> Looks like man ethtool, right? >> >> On Mon, Feb 9, 2015 at 5:49 PM, DJ Root wrote: >>> Adam, >>> >>> What you describe here is a fairly straightforward match/action rule - in your case you want perform some action when you receive dst host 10.8.0.56 / dst port 53 / src host X / src port Y traffic - right? This operation can be done on an intelligent NIC card without any interaction with Bro (or BPF for that matter). >>> >>> If you would like more information on how we could help solve this problem, please email me privately. >>> >>> Regards, >>> DJ Root >>>> On Feb 9, 2015, at 9:19 AM, Seth Hall wrote: >>>> >>>> >>>>> On Feb 8, 2015, at 7:53 PM, Adam Hall wrote: >>>>> >>>>> 1423442632.139980 bro (ip or not ip) and (not (dst host 10.8.0.85 and dst port 53)) T T >>>>> The only way I have been able to successfully get this to work is by defining only "host" or "port", I have not been able to get this to be successful using a "src host", "dst host", "src port", or "dst port". >>>>> >>>>> This creates a problem to the point it's almost unusable to me as I cannot ignore all traffic for "host 10.8.0.85 and port 53". >>>> >>>> It?s unlikely that you are ever going to want to use the ?src? or ?dst? modifiers in filters meant for Bro. Bro is intended to look at both directions of traffic to successfully analyze connections and you are intrinsically breaking that when you use src or dst. >>>> >>>> What is the end result you?re trying to get to? You just don?t want to see dns traffic involving host 10.8.0.85? >>>> >>>> The problem here is that there is a disjoint in the semantics of BPF and the typical semantics of Bro. Bro operates on connections (and flows to a slightly lesser degree currently). BPF is completely oriented around packets. It causes these little confusions unfortunately. >>>> >>>> .Seth >>>> >>>> -- >>>> Seth Hall >>>> International Computer Science Institute >>>> (Bro) because everyone has a network >>>> http://www.bro.org/ >>>> >>>> >>>> _______________________________________________ >>>> Bro mailing list >>>> bro at bro-ids.org >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From dj.root at netronome.com Mon Feb 9 10:30:58 2015 From: dj.root at netronome.com (DJ Root) Date: Mon, 9 Feb 2015 13:30:58 -0500 Subject: [Bro] BPF Filter Help In-Reply-To: References: <795783142.1660128.1423443191393.JavaMail.yahoo@mail.yahoo.com> <45CA62D7-4066-4611-B946-E13C4751E197@icir.org> Message-ID: <9DF4A6E9-A88B-4EAC-B7B6-7117D20565F2@netronome.com> No. We design and develop our own ASIC (Flow Processor). It is the NFP32xx and NFP6xxx. We use Intel as our foundry. Regards, DJ > On Feb 9, 2015, at 1:28 PM, Micha? Purzy?ski wrote: > > Do you use X520 chip by accident? > > On Mon, Feb 9, 2015 at 7:20 PM, DJ Root wrote: >> Michal, >> >> No, there is not. However, I come from the vendor side and, therefore, don?t want to disrupt the integrity of a technology mailing list. That said we have Bro running in our lab, so our claims can be supported by real data and demos. >> >> As far as the question below is concerned, we (Netronome) have intelligent NICs and software to do match / action operations in hardware. We can set-up 5-tuple filtering in hardware which can address Adam?s problem. Result - BPF is off-loaded from the CPU; Bro is not changed, but now has more CPU cycles to process flows. >> >> Regards, >> DJ Root >> >> DJ Root >> Director of Sales, Americas East and EMEA >> Netronome, Inc. >> (617)686-0253 >> >> >> >> >>> On Feb 9, 2015, at 11:57 AM, Micha? Purzy?ski wrote: >>> >>> Is there any reason why you cannot share this kind of information on >>> the list, so everyone can benefit? >>> >>> Looks like man ethtool, right? >>> >>> On Mon, Feb 9, 2015 at 5:49 PM, DJ Root wrote: >>>> Adam, >>>> >>>> What you describe here is a fairly straightforward match/action rule - in your case you want perform some action when you receive dst host 10.8.0.56 / dst port 53 / src host X / src port Y traffic - right? This operation can be done on an intelligent NIC card without any interaction with Bro (or BPF for that matter). >>>> >>>> If you would like more information on how we could help solve this problem, please email me privately. >>>> >>>> Regards, >>>> DJ Root >>>>> On Feb 9, 2015, at 9:19 AM, Seth Hall wrote: >>>>> >>>>> >>>>>> On Feb 8, 2015, at 7:53 PM, Adam Hall wrote: >>>>>> >>>>>> 1423442632.139980 bro (ip or not ip) and (not (dst host 10.8.0.85 and dst port 53)) T T >>>>>> The only way I have been able to successfully get this to work is by defining only "host" or "port", I have not been able to get this to be successful using a "src host", "dst host", "src port", or "dst port". >>>>>> >>>>>> This creates a problem to the point it's almost unusable to me as I cannot ignore all traffic for "host 10.8.0.85 and port 53". >>>>> >>>>> It?s unlikely that you are ever going to want to use the ?src? or ?dst? modifiers in filters meant for Bro. Bro is intended to look at both directions of traffic to successfully analyze connections and you are intrinsically breaking that when you use src or dst. >>>>> >>>>> What is the end result you?re trying to get to? You just don?t want to see dns traffic involving host 10.8.0.85? >>>>> >>>>> The problem here is that there is a disjoint in the semantics of BPF and the typical semantics of Bro. Bro operates on connections (and flows to a slightly lesser degree currently). BPF is completely oriented around packets. It causes these little confusions unfortunately. >>>>> >>>>> .Seth >>>>> >>>>> -- >>>>> Seth Hall >>>>> International Computer Science Institute >>>>> (Bro) because everyone has a network >>>>> http://www.bro.org/ >>>>> >>>>> >>>>> _______________________________________________ >>>>> Bro mailing list >>>>> bro at bro-ids.org >>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>>> >>>> >>>> _______________________________________________ >>>> Bro mailing list >>>> bro at bro-ids.org >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> From abhall1 at yahoo.com Tue Feb 10 13:11:48 2015 From: abhall1 at yahoo.com (abhall1) Date: Tue, 10 Feb 2015 16:11:48 -0500 Subject: [Bro] Bro Digest, Vol 106, Issue 14 Message-ID: To Seth Hall, ? ? ?Thank you for the response. ?I still wish I could filter traffic that way, but I do see the logic in your reasoning. ?We decided ultimately to stick with the filter for that host and port 53. Thank you for your time and help! Adam hall Sent via the Samsung Galaxy Note? 4, an AT&T 4G LTE smartphone -------- Original message -------- From: bro-request at bro.org Date: 02/09/2015 3:00 PM (GMT-05:00) To: bro at bro.org Subject: Bro Digest, Vol 106, Issue 14 Send Bro mailing list submissions to bro at bro.org To subscribe or unsubscribe via the World Wide Web, visit http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro or, via email, send a message with subject or body 'help' to bro-request at bro.org You can reach the person managing the list at bro-owner at bro.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Bro digest..." Today's Topics: 1. Re: BPF Filter Help (Micha? Purzy?ski) 2. Re: BPF Filter Help (DJ Root) ---------------------------------------------------------------------- Message: 1 Date: Mon, 9 Feb 2015 19:28:25 +0100 From: Micha? Purzy?ski Subject: Re: [Bro] BPF Filter Help To: DJ Root Cc: Adam Hall , "Bro at bro.org" Message-ID: Content-Type: text/plain; charset=UTF-8 Do you use X520 chip by accident? On Mon, Feb 9, 2015 at 7:20 PM, DJ Root wrote: > Michal, > > No, there is not. However, I come from the vendor side and, therefore, don?t want to disrupt the integrity of a technology mailing list. That said we have Bro running in our lab, so our claims can be supported by real data and demos. > > As far as the question below is concerned, we (Netronome) have intelligent NICs and software to do match / action operations in hardware. We can set-up 5-tuple filtering in hardware which can address Adam?s problem. Result - BPF is off-loaded from the CPU; Bro is not changed, but now has more CPU cycles to process flows. > > Regards, > DJ Root > > DJ Root > Director of Sales, Americas East and EMEA > Netronome, Inc. > (617)686-0253 > > > > >> On Feb 9, 2015, at 11:57 AM, Micha? Purzy?ski wrote: >> >> Is there any reason why you cannot share this kind of information on >> the list, so everyone can benefit? >> >> Looks like man ethtool, right? >> >> On Mon, Feb 9, 2015 at 5:49 PM, DJ Root wrote: >>> Adam, >>> >>> What you describe here is a fairly straightforward match/action rule - in your case you want perform some action when you receive dst host 10.8.0.56 / dst port 53 / src host X / src port Y traffic - right? This operation can be done on an intelligent NIC card without any interaction with Bro (or BPF for that matter). >>> >>> If you would like more information on how we could help solve this problem, please email me privately. >>> >>> Regards, >>> DJ Root >>>> On Feb 9, 2015, at 9:19 AM, Seth Hall wrote: >>>> >>>> >>>>> On Feb 8, 2015, at 7:53 PM, Adam Hall wrote: >>>>> >>>>> 1423442632.139980 bro (ip or not ip) and (not (dst host 10.8.0.85 and dst port 53)) T T >>>>> The only way I have been able to successfully get this to work is by defining only "host" or "port", I have not been able to get this to be successful using a "src host", "dst host", "src port", or "dst port". >>>>> >>>>> This creates a problem to the point it's almost unusable to me as I cannot ignore all traffic for "host 10.8.0.85 and port 53". >>>> >>>> It?s unlikely that you are ever going to want to use the ?src? or ?dst? modifiers in filters meant for Bro. Bro is intended to look at both directions of traffic to successfully analyze connections and you are intrinsically breaking that when you use src or dst. >>>> >>>> What is the end result you?re trying to get to? You just don?t want to see dns traffic involving host 10.8.0.85? >>>> >>>> The problem here is that there is a disjoint in the semantics of BPF and the typical semantics of Bro. Bro operates on connections (and flows to a slightly lesser degree currently). BPF is completely oriented around packets. It causes these little confusions unfortunately. >>>> >>>> .Seth >>>> >>>> -- >>>> Seth Hall >>>> International Computer Science Institute >>>> (Bro) because everyone has a network >>>> http://www.bro.org/ >>>> >>>> >>>> _______________________________________________ >>>> Bro mailing list >>>> bro at bro-ids.org >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > ------------------------------ Message: 2 Date: Mon, 9 Feb 2015 13:30:58 -0500 From: DJ Root Subject: Re: [Bro] BPF Filter Help To: Micha? Purzy?ski Cc: Adam Hall , "Bro at bro.org" Message-ID: <9DF4A6E9-A88B-4EAC-B7B6-7117D20565F2 at netronome.com> Content-Type: text/plain; charset=utf-8 No. We design and develop our own ASIC (Flow Processor). It is the NFP32xx and NFP6xxx. We use Intel as our foundry. Regards, DJ > On Feb 9, 2015, at 1:28 PM, Micha? Purzy?ski wrote: > > Do you use X520 chip by accident? > > On Mon, Feb 9, 2015 at 7:20 PM, DJ Root wrote: >> Michal, >> >> No, there is not. However, I come from the vendor side and, therefore, don?t want to disrupt the integrity of a technology mailing list. That said we have Bro running in our lab, so our claims can be supported by real data and demos. >> >> As far as the question below is concerned, we (Netronome) have intelligent NICs and software to do match / action operations in hardware. We can set-up 5-tuple filtering in hardware which can address Adam?s problem. Result - BPF is off-loaded from the CPU; Bro is not changed, but now has more CPU cycles to process flows. >> >> Regards, >> DJ Root >> >> DJ Root >> Director of Sales, Americas East and EMEA >> Netronome, Inc. >> (617)686-0253 >> >> >> >> >>> On Feb 9, 2015, at 11:57 AM, Micha? Purzy?ski wrote: >>> >>> Is there any reason why you cannot share this kind of information on >>> the list, so everyone can benefit? >>> >>> Looks like man ethtool, right? >>> >>> On Mon, Feb 9, 2015 at 5:49 PM, DJ Root wrote: >>>> Adam, >>>> >>>> What you describe here is a fairly straightforward match/action rule - in your case you want perform some action when you receive dst host 10.8.0.56 / dst port 53 / src host X / src port Y traffic - right? This operation can be done on an intelligent NIC card without any interaction with Bro (or BPF for that matter). >>>> >>>> If you would like more information on how we could help solve this problem, please email me privately. >>>> >>>> Regards, >>>> DJ Root >>>>> On Feb 9, 2015, at 9:19 AM, Seth Hall wrote: >>>>> >>>>> >>>>>> On Feb 8, 2015, at 7:53 PM, Adam Hall wrote: >>>>>> >>>>>> 1423442632.139980 bro (ip or not ip) and (not (dst host 10.8.0.85 and dst port 53)) T T >>>>>> The only way I have been able to successfully get this to work is by defining only "host" or "port", I have not been able to get this to be successful using a "src host", "dst host", "src port", or "dst port". >>>>>> >>>>>> This creates a problem to the point it's almost unusable to me as I cannot ignore all traffic for "host 10.8.0.85 and port 53". >>>>> >>>>> It?s unlikely that you are ever going to want to use the ?src? or ?dst? modifiers in filters meant for Bro. Bro is intended to look at both directions of traffic to successfully analyze connections and you are intrinsically breaking that when you use src or dst. >>>>> >>>>> What is the end result you?re trying to get to? You just don?t want to see dns traffic involving host 10.8.0.85? >>>>> >>>>> The problem here is that there is a disjoint in the semantics of BPF and the typical semantics of Bro. Bro operates on connections (and flows to a slightly lesser degree currently). BPF is completely oriented around packets. It causes these little confusions unfortunately. >>>>> >>>>> .Seth >>>>> >>>>> -- >>>>> Seth Hall >>>>> International Computer Science Institute >>>>> (Bro) because everyone has a network >>>>> http://www.bro.org/ >>>>> >>>>> >>>>> _______________________________________________ >>>>> Bro mailing list >>>>> bro at bro-ids.org >>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>>> >>>> >>>> _______________________________________________ >>>> Bro mailing list >>>> bro at bro-ids.org >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> ------------------------------ _______________________________________________ Bro mailing list Bro at bro.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro End of Bro Digest, Vol 106, Issue 14 ************************************ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150210/33514fa3/attachment-0001.html From mwsong at imtl.skku.ac.kr Tue Feb 10 21:23:32 2015 From: mwsong at imtl.skku.ac.kr (mwsong) Date: Wed, 11 Feb 2015 14:23:32 +0900 Subject: [Bro] =?utf-8?q?Questions_abot_new=5Fpacket?= Message-ID: <328225fe55f87ad12ac3db591c7f95fa@imtl.skku.ac.kr> Hi i have questions about new_packet event 1) how can I get all the packet payload bro sees? - My result - new_packet give only packet header. - packet_contents give transport layer payload - Both packets are not matched - tcp_packets does not return about http (i guess it divided) - So i want to know - How can I get full header and body of packets? - Is there any way to packet mirror? From michalpurzynski1 at gmail.com Wed Feb 11 07:14:30 2015 From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=) Date: Wed, 11 Feb 2015 16:14:30 +0100 Subject: [Bro] zbalance_ipc with multiple applications and Bro Message-ID: Hi. I'm trying to start Bro and Suricata on one sensor, using the pf_ring ZC, like this zbalance_ipc -i eth5 -c 99 -n 4,4 -m 1 where 99 is the cluster ID and -n , creates separate rings for each application. So far so good. I should tell Bro to somehow bind to the zc:99 at 4, zc:99 at 5, zc:99 at 6, zc:99 at 7 interfaces. How can I do it? Using zc:99 at 4 (AKA base, and let it increment automatically) does not work fatal error: /opt/bro/bin/bro: problem with interface zc:99 at 4 - pcap_open_live: zc:99 at 4: No such device exists (SIOCGIFHWADDR: No such device) Same for just zc:99 and not a surprise, Bro somehow needs to open sub-interfaces 4-7. Is it even supported? From apumphrey at ivsec.com Wed Feb 11 08:31:29 2015 From: apumphrey at ivsec.com (Adam Pumphrey) Date: Wed, 11 Feb 2015 11:31:29 -0500 Subject: [Bro] zbalance_ipc with multiple applications and Bro In-Reply-To: References: Message-ID: <76ACD60A-3478-491F-9215-51D152484870@ivsec.com> You probably need to take a look at the PFRINGFirstAppInstance in broctl.cfg, it defaults to 0. If you?re looking to use the second app instance created by zbalance_ipc you?ll need to set that option to 4. Also make sure the lb_method and lb_procs are set appropriately in node.cfg file, for example: interface=zc:99 lb_method=pf_ring lb_procs=4 # should be equivalent to the number of instances per ?ring' If you really want to use zero-copy you need to add the prefix ?zc:? to the physical interface name; e.g. zbalance_ipc -i zc:eth5. There are other pre-req?s for that to work, like configuring huge memory pages and installing the pf_ring-aware ZC driver. I?ve been testing with ZC also but having issues with Bro reporting increased packet loss rates as soon I enable a configuration like this. Not sure if this is a hashing mode conflict, NIC/driver configuration issue or what.. I?d be interested to hear about your (or anyone else?s) results with such a setup. Adam > On Feb 11, 2015, at 10:14 AM, Micha? Purzy?ski wrote: > > Hi. > > I'm trying to start Bro and Suricata on one sensor, using the pf_ring > ZC, like this > > zbalance_ipc -i eth5 -c 99 -n 4,4 -m 1 > > where 99 is the cluster ID and -n , creates separate rings > for each application. So far so good. > > I should tell Bro to somehow bind to the zc:99 at 4, zc:99 at 5, zc:99 at 6, > zc:99 at 7 interfaces. How can I do it? > > Using zc:99 at 4 (AKA base, and let it increment automatically) does not work > > fatal error: /opt/bro/bin/bro: problem with interface zc:99 at 4 - > pcap_open_live: zc:99 at 4: No such device exists (SIOCGIFHWADDR: No such > device) > > Same for just zc:99 and not a surprise, Bro somehow needs to open > sub-interfaces 4-7. > > Is it even supported? > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From dj.root at netronome.com Wed Feb 11 08:45:41 2015 From: dj.root at netronome.com (DJ Root) Date: Wed, 11 Feb 2015 11:45:41 -0500 Subject: [Bro] zbalance_ipc with multiple applications and Bro In-Reply-To: References: Message-ID: Hi Michal, It may be supported, but we have tested and proven similar functionality in hardware. Our hardware and software can bind specific instances of Bro (or Suricata for that matter) onto host cores - something we call flow affinity. Furthermore, those flows are load balanced anyway the user wants them. Example config: We compile both Bro and Suricata against our pcap libraries so that they each recognize our network interface nomenclature. In the case for ?Bro?, we edit the ?/usr/local/bro/etc/node.cfg" file to add the interface bindings and cpu pinning for each worker thread. See below. We then use ?broctl? to start Bro processing. [worker-1] type=worker host=172.24.3.9 pin_cpus=0 interface=nfe0.1.0 [worker-2] type=worker host=172.24.3.9 pin_cpus=1 interface=nfe0.1.1 [worker-3] type=worker host=172.24.3.9 pin_cpus=2 interface=nfe0.1.2 [worker-4] type=worker host=172.24.3.9 pin_cpus=3 interface=nfe0.1.3 Note nfe = Netronome Flow Engine Regards, DJ > On Feb 11, 2015, at 10:14 AM, Micha? Purzy?ski wrote: > > Hi. > > I'm trying to start Bro and Suricata on one sensor, using the pf_ring > ZC, like this > > zbalance_ipc -i eth5 -c 99 -n 4,4 -m 1 > > where 99 is the cluster ID and -n , creates separate rings > for each application. So far so good. > > I should tell Bro to somehow bind to the zc:99 at 4, zc:99 at 5, zc:99 at 6, > zc:99 at 7 interfaces. How can I do it? > > Using zc:99 at 4 (AKA base, and let it increment automatically) does not work > > fatal error: /opt/bro/bin/bro: problem with interface zc:99 at 4 - > pcap_open_live: zc:99 at 4: No such device exists (SIOCGIFHWADDR: No such > device) > > Same for just zc:99 and not a surprise, Bro somehow needs to open > sub-interfaces 4-7. > > Is it even supported? > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From john.rote at shadownetworks.com Wed Feb 11 09:18:10 2015 From: john.rote at shadownetworks.com (John Rote) Date: Wed, 11 Feb 2015 09:18:10 -0800 Subject: [Bro] Removing GeoIP Location from Bro Message-ID: All, I am trying to remove Geo IP capability from Bro and have most of it removed but when I attempt to remove the global variable, the bro check fails. What are the steps for removing Geo IP from Bro? Thanks John -- John Rote Director, Security Architecture - Shadow Labs Shadow Networks john.rote at shadownetworks.com 408-242-9688 From michalpurzynski1 at gmail.com Wed Feb 11 09:18:09 2015 From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=) Date: Wed, 11 Feb 2015 18:18:09 +0100 Subject: [Bro] zbalance_ipc with multiple applications and Bro In-Reply-To: <76ACD60A-3478-491F-9215-51D152484870@ivsec.com> References: <76ACD60A-3478-491F-9215-51D152484870@ivsec.com> Message-ID: I'm clearly doing something wrong. pfring-svn-latest/userland/examples_zc ? ./zbalance_ipc -i zc:eth5 -c 99 -n 4,4 -m 1 [nsm1-sfo-eth5] type=worker host=10.251.75.9 interface=zc:99 lb_method=pf_ring lb_procs=4 grep PFRINGFirstAppInstance broctl.cfg PFRINGFirstAppInstance = 4 fatal error: /opt/bro/bin/bro: problem with interface zc:99 - pcap_open_live: zc:99: No such device exists (SIOCGIFHWADDR: No such device) And yeah, Bro is compiled against the pf_ring libpcap. On Wed, Feb 11, 2015 at 5:31 PM, Adam Pumphrey wrote: > You probably need to take a look at the PFRINGFirstAppInstance in broctl.cfg, it defaults to 0. If you?re looking to use the second app instance created by zbalance_ipc you?ll need to set that option to 4. > > Also make sure the lb_method and lb_procs are set appropriately in node.cfg file, for example: > > interface=zc:99 > lb_method=pf_ring > lb_procs=4 # should be equivalent to the number of instances per ?ring' > > If you really want to use zero-copy you need to add the prefix ?zc:? to the physical interface name; e.g. zbalance_ipc -i zc:eth5. There are other pre-req?s for that to work, like configuring huge memory pages and installing the pf_ring-aware ZC driver. > > I?ve been testing with ZC also but having issues with Bro reporting increased packet loss rates as soon I enable a configuration like this. Not sure if this is a hashing mode conflict, NIC/driver configuration issue or what.. I?d be interested to hear about your (or anyone else?s) results with such a setup. > > Adam > >> On Feb 11, 2015, at 10:14 AM, Micha? Purzy?ski wrote: >> >> Hi. >> >> I'm trying to start Bro and Suricata on one sensor, using the pf_ring >> ZC, like this >> >> zbalance_ipc -i eth5 -c 99 -n 4,4 -m 1 >> >> where 99 is the cluster ID and -n , creates separate rings >> for each application. So far so good. >> >> I should tell Bro to somehow bind to the zc:99 at 4, zc:99 at 5, zc:99 at 6, >> zc:99 at 7 interfaces. How can I do it? >> >> Using zc:99 at 4 (AKA base, and let it increment automatically) does not work >> >> fatal error: /opt/bro/bin/bro: problem with interface zc:99 at 4 - >> pcap_open_live: zc:99 at 4: No such device exists (SIOCGIFHWADDR: No such >> device) >> >> Same for just zc:99 and not a surprise, Bro somehow needs to open >> sub-interfaces 4-7. >> >> Is it even supported? >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From michalpurzynski1 at gmail.com Wed Feb 11 09:21:09 2015 From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=) Date: Wed, 11 Feb 2015 18:21:09 +0100 Subject: [Bro] zbalance_ipc with multiple applications and Bro In-Reply-To: References: Message-ID: Hello, vendor. Thank you a lot for sending email that says "I have no answer to your question, but here is what I can sell you" Guess what. You won't. Ever. From anthony.kasza at gmail.com Wed Feb 11 09:33:54 2015 From: anthony.kasza at gmail.com (anthony kasza) Date: Wed, 11 Feb 2015 09:33:54 -0800 Subject: [Bro] Removing GeoIP Location from Bro In-Reply-To: References: Message-ID: You'll need to remove references to it in a CPP and header file (I forget which ones right now bit will get back to you). You need to remove the functions that use the geoip record type from the bifs. You'll also need to remove ant references to the geoip functions and the geoip record type definition from script land. -AK On Feb 11, 2015 9:25 AM, "John Rote" wrote: > All, > > I am trying to remove Geo IP capability from Bro and have most of it > removed but when I attempt to remove the global variable, the bro > check fails. > > What are the steps for removing Geo IP from Bro? > > Thanks > John > > -- > John Rote > Director, Security Architecture - Shadow Labs > Shadow Networks > john.rote at shadownetworks.com > 408-242-9688 > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150211/e870a5e7/attachment-0001.html From timo.seppala at gmail.com Wed Feb 11 10:20:43 2015 From: timo.seppala at gmail.com (=?UTF-8?B?VGltbyBTZXBww6Rsw6Q=?=) Date: Wed, 11 Feb 2015 20:20:43 +0200 Subject: [Bro] E-mail address removal from the mailing list Message-ID: hey, would you please remove my email address from the distribution list or, alternatively, to tell you how I can get myself out of the address on the distribution list. Best regards Timo Sepp?l? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150211/cfd3a4b9/attachment.html From jonschipp at gmail.com Wed Feb 11 10:50:23 2015 From: jonschipp at gmail.com (Jon Schipp) Date: Wed, 11 Feb 2015 12:50:23 -0600 Subject: [Bro] E-mail address removal from the mailing list In-Reply-To: References: Message-ID: There's an unsubscribe option at http://mailman.icsi.berkeley.edu/mailman/listinfo/bro On Wed, Feb 11, 2015 at 12:20 PM, Timo Sepp?l? wrote: > hey, > would you please remove my email address from the distribution list or, > alternatively, to tell you how I can get myself out of the address on the > distribution list. > > Best regards > Timo Sepp?l? > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Jon Schipp, jonschipp.com, sickbits.net, opennsm.ncsa.illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150211/8d759bd6/attachment.html From apumphrey at ivsec.com Wed Feb 11 12:03:23 2015 From: apumphrey at ivsec.com (Adam Pumphrey) Date: Wed, 11 Feb 2015 15:03:23 -0500 Subject: [Bro] zbalance_ipc with multiple applications and Bro In-Reply-To: References: <76ACD60A-3478-491F-9215-51D152484870@ivsec.com> Message-ID: Your Bro config looks like it should work. From what I?ve seen that usually indicates an issue with pf_ring; possibly that zbalance_ipc is failing to run? A couple other things to check on the pf_ring side, all of which applies to your worker nodes. Sorry if any of this obvious, just throwing out ideas: - pf_ring kernel module installed - pf_ring-aware ZC NIC driver installed and in use by the physical interface (ethtool -i) - ZC license installed - huge memory pages configured If successful zbalance_ipc should output (when not in daemon mode or stdout/stderr redirected) something like this, followed by traffic collection stats: Starting balancer with 8 consumer queues.. You can now attach to the balancer your application instances as follows: Application 0 pfcount -i zc:99 at 0 pfcount -i zc:99 at 1 pfcount -i zc:99 at 2 pfcount -i zc:99 at 3 Application 1 pfcount -i zc:99 at 4 pfcount -i zc:99 at 5 pfcount -i zc:99 at 6 pfcount -i zc:99 at 7 Once zbalance_ipc is running you can use zcount_ipc as another way to validate what zbalance is doing. If you can run zcount_ipc and get packets from each of the app instances, your Bro config should work too. Adam > On Feb 11, 2015, at 12:18 PM, Micha? Purzy?ski wrote: > > I'm clearly doing something wrong. > > pfring-svn-latest/userland/examples_zc ? ./zbalance_ipc -i zc:eth5 -c > 99 -n 4,4 -m 1 > > > [nsm1-sfo-eth5] > type=worker > host=10.251.75.9 > interface=zc:99 > lb_method=pf_ring > lb_procs=4 > > grep PFRINGFirstAppInstance broctl.cfg > > PFRINGFirstAppInstance = 4 > > > fatal error: /opt/bro/bin/bro: problem with interface zc:99 - > pcap_open_live: zc:99: No such device exists (SIOCGIFHWADDR: No such > device) > > And yeah, Bro is compiled against the pf_ring libpcap. > > On Wed, Feb 11, 2015 at 5:31 PM, Adam Pumphrey wrote: >> You probably need to take a look at the PFRINGFirstAppInstance in broctl.cfg, it defaults to 0. If you?re looking to use the second app instance created by zbalance_ipc you?ll need to set that option to 4. >> >> Also make sure the lb_method and lb_procs are set appropriately in node.cfg file, for example: >> >> interface=zc:99 >> lb_method=pf_ring >> lb_procs=4 # should be equivalent to the number of instances per ?ring' >> >> If you really want to use zero-copy you need to add the prefix ?zc:? to the physical interface name; e.g. zbalance_ipc -i zc:eth5. There are other pre-req?s for that to work, like configuring huge memory pages and installing the pf_ring-aware ZC driver. >> >> I?ve been testing with ZC also but having issues with Bro reporting increased packet loss rates as soon I enable a configuration like this. Not sure if this is a hashing mode conflict, NIC/driver configuration issue or what.. I?d be interested to hear about your (or anyone else?s) results with such a setup. >> >> Adam >> >>> On Feb 11, 2015, at 10:14 AM, Micha? Purzy?ski wrote: >>> >>> Hi. >>> >>> I'm trying to start Bro and Suricata on one sensor, using the pf_ring >>> ZC, like this >>> >>> zbalance_ipc -i eth5 -c 99 -n 4,4 -m 1 >>> >>> where 99 is the cluster ID and -n , creates separate rings >>> for each application. So far so good. >>> >>> I should tell Bro to somehow bind to the zc:99 at 4, zc:99 at 5, zc:99 at 6, >>> zc:99 at 7 interfaces. How can I do it? >>> >>> Using zc:99 at 4 (AKA base, and let it increment automatically) does not work >>> >>> fatal error: /opt/bro/bin/bro: problem with interface zc:99 at 4 - >>> pcap_open_live: zc:99 at 4: No such device exists (SIOCGIFHWADDR: No such >>> device) >>> >>> Same for just zc:99 and not a surprise, Bro somehow needs to open >>> sub-interfaces 4-7. >>> >>> Is it even supported? >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150211/5a2a58fa/attachment.html From seth at icir.org Wed Feb 11 20:08:29 2015 From: seth at icir.org (Seth Hall) Date: Wed, 11 Feb 2015 23:08:29 -0500 Subject: [Bro] zbalance_ipc with multiple applications and Bro In-Reply-To: References: Message-ID: > On Feb 11, 2015, at 10:14 AM, Micha? Purzy?ski wrote: > > Same for just zc:99 and not a surprise, Bro somehow needs to open > sub-interfaces 4-7. > > Is it even supported? It is actually. :) In node.cfg, you would config a worker like this? [worker-x] type=worker host=1.2.3.4 interface=zc:99 lb_method=pf_ring lb_procs=4 In broctl.cfg, you?d set this? pfringfirstappinstance=4 That should make it work. Unfortunately pfringfirstappinstance is a global variable for broctl, but it was easier when I was implementing it and I would generally expect people to be running the same or substantially similar zbalance_ipc configurations everywhere. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From michalpurzynski1 at gmail.com Thu Feb 12 06:49:16 2015 From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=) Date: Thu, 12 Feb 2015 15:49:16 +0100 Subject: [Bro] zbalance_ipc with multiple applications and Bro In-Reply-To: References: Message-ID: Thanks a lot - it seems that my own build of Bro has been linked to the system libpcap instead of the pf_ring one. I'm fighting with that now, will update this thread with observations, once I get it working. On Thu, Feb 12, 2015 at 5:08 AM, Seth Hall wrote: > >> On Feb 11, 2015, at 10:14 AM, Micha? Purzy?ski wrote: >> >> Same for just zc:99 and not a surprise, Bro somehow needs to open >> sub-interfaces 4-7. >> >> Is it even supported? > > It is actually. :) > > In node.cfg, you would config a worker like this? > > [worker-x] > type=worker > host=1.2.3.4 > interface=zc:99 > lb_method=pf_ring > lb_procs=4 > > In broctl.cfg, you?d set this? > > pfringfirstappinstance=4 > > That should make it work. Unfortunately pfringfirstappinstance is a global variable for broctl, but it was easier when I was implementing it and I would generally expect people to be running the same or substantially similar zbalance_ipc configurations everywhere. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > From michalpurzynski1 at gmail.com Thu Feb 12 07:14:22 2015 From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=) Date: Thu, 12 Feb 2015 16:14:22 +0100 Subject: [Bro] Bro 2.3.2 and pf_ring 6.0.2 Message-ID: When trying to build Bro 2.3.2 with libpcap from pf_ring 6.0.2 (./configure --with-pcap=/usr/local/lib, where libpcap-1.1.1-ring is installed) I get errors: -- Looking for include files HAVE_PCAP_INT_H -- Looking for include files HAVE_PCAP_INT_H - not found. -- Looking for pcap_freecode -- Looking for pcap_freecode - not found -- No implementation for pcap_freecode() -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_HAS_ERROR_PARAMETER -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_HAS_ERROR_PARAMETER - Failed -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_NO_ERROR_PARAMETER -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_NO_ERROR_PARAMETER - Failed CMake Error at cmake/PCAPTests.cmake:58 (message): Can't determine if pcap_compile_nopcap takes an error parameter Call Stack (most recent call first): CMakeLists.txt:176 (include) -- Configuring incomplete, errors occurred! There is no other libpcap on this system. I have just tried on a clean and freshly installed VM. What am I doing wrong? I'd like to link Bro with pcap pf_ring, not the system one. grep PCAP build/CMakeCache.txt PCAP_INCLUDE_DIR:PATH=/usr/local/include PCAP_LIBRARY:FILEPATH=/usr/local/lib/libpcap.so PCAP_ROOT_DIR:PATH=/usr/local/lib //Details about finding PCAP FIND_PACKAGE_MESSAGE_DETAILS_PCAP:INTERNAL=[/usr/local/lib/libpcap.so][/usr/local/include][v()] HAVE_LIBPCAP_PCAP_FREECODE:INTERNAL= //Have includes HAVE_PCAP_INT_H HAVE_PCAP_INT_H:INTERNAL= //Test LIBPCAP_PCAP_COMPILE_NOPCAP_HAS_ERROR_PARAMETER LIBPCAP_PCAP_COMPILE_NOPCAP_HAS_ERROR_PARAMETER:INTERNAL= //Test LIBPCAP_PCAP_COMPILE_NOPCAP_NO_ERROR_PARAMETER LIBPCAP_PCAP_COMPILE_NOPCAP_NO_ERROR_PARAMETER:INTERNAL= //ADVANCED property for variable: PCAP_INCLUDE_DIR PCAP_INCLUDE_DIR-ADVANCED:INTERNAL=1 //ADVANCED property for variable: PCAP_LIBRARY PCAP_LIBRARY-ADVANCED:INTERNAL=1 //Test PCAP_LINKS_SOLO PCAP_LINKS_SOLO:INTERNAL=1 //ADVANCED property for variable: PCAP_ROOT_DIR PCAP_ROOT_DIR-ADVANCED:INTERNAL=1 From jsiwek at illinois.edu Thu Feb 12 07:35:26 2015 From: jsiwek at illinois.edu (Siwek, Jon) Date: Thu, 12 Feb 2015 15:35:26 +0000 Subject: [Bro] Bro 2.3.2 and pf_ring 6.0.2 In-Reply-To: References: Message-ID: > On Feb 12, 2015, at 9:14 AM, Micha? Purzy?ski wrote: > > When trying to build Bro 2.3.2 with libpcap from pf_ring 6.0.2 > (./configure --with-pcap=/usr/local/lib, where libpcap-1.1.1-ring is > installed) I get errors: > > -- Looking for include files HAVE_PCAP_INT_H > -- Looking for include files HAVE_PCAP_INT_H - not found. > -- Looking for pcap_freecode > -- Looking for pcap_freecode - not found > -- No implementation for pcap_freecode() > -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_HAS_ERROR_PARAMETER > -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_HAS_ERROR_PARAMETER - Failed > -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_NO_ERROR_PARAMETER > -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_NO_ERROR_PARAMETER - Failed > CMake Error at cmake/PCAPTests.cmake:58 (message): > Can't determine if pcap_compile_nopcap takes an error parameter > Call Stack (most recent call first): > CMakeLists.txt:176 (include) > -- Configuring incomplete, errors occurred! Can you send the contents of build/CMakeFiles/CMakeError.log ? - Jon From seth at icir.org Thu Feb 12 08:05:02 2015 From: seth at icir.org (Seth Hall) Date: Thu, 12 Feb 2015 11:05:02 -0500 Subject: [Bro] Bro 2.3.2 and pf_ring 6.0.2 In-Reply-To: References: Message-ID: <0A30B0E2-82BF-407B-A6F2-EBC367BE39A1@icir.org> > On Feb 12, 2015, at 10:14 AM, Micha? Purzy?ski wrote: > > (./configure --with-pcap=/usr/local/lib, where libpcap-1.1.1-ring is I think you just want /usr/local/ for that argument. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From himself at louruppert.com Thu Feb 12 08:10:30 2015 From: himself at louruppert.com (Lou Ruppert) Date: Thu, 12 Feb 2015 16:10:30 +0000 Subject: [Bro] Bro 2.3.2 and pf_ring 6.0.2 In-Reply-To: References: Message-ID: <54DCD076.5020807@louruppert.com> Siwek, Jon: > >> On Feb 12, 2015, at 9:14 AM, Micha? Purzy?ski wrote: >> >> When trying to build Bro 2.3.2 with libpcap from pf_ring 6.0.2 >> (./configure --with-pcap=/usr/local/lib, where libpcap-1.1.1-ring is >> installed) I get errors: >> >> -- Looking for include files HAVE_PCAP_INT_H >> -- Looking for include files HAVE_PCAP_INT_H - not found. >> -- Looking for pcap_freecode >> -- Looking for pcap_freecode - not found >> -- No implementation for pcap_freecode() >> -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_HAS_ERROR_PARAMETER >> -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_HAS_ERROR_PARAMETER - Failed >> -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_NO_ERROR_PARAMETER >> -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_NO_ERROR_PARAMETER - Failed >> CMake Error at cmake/PCAPTests.cmake:58 (message): >> Can't determine if pcap_compile_nopcap takes an error parameter >> Call Stack (most recent call first): >> CMakeLists.txt:176 (include) >> -- Configuring incomplete, errors occurred! > > Can you send the contents of build/CMakeFiles/CMakeError.log ? I wish I'd thought of that. Instead I kludged a quick solution: LDFLAGS="-Wl,--no-as-needed -lrt" ./configure I'm not sure why that bit of ugliness helps, but it got me up and running again. -lou -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: OpenPGP digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150212/a9776f6e/attachment.bin From michalpurzynski1 at gmail.com Thu Feb 12 08:11:16 2015 From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=) Date: Thu, 12 Feb 2015 17:11:16 +0100 Subject: [Bro] Bro 2.3.2 and pf_ring 6.0.2 In-Reply-To: References: Message-ID: /usr/bin/gcc -Wall -Wno-unused -DCHECK_FUNCTION_EXISTS=pcap_get_pfring_id CMakeFiles/cmTryCompileExec.dir/CheckFunctionExists.c.o -o cmTryCompileExec -rdynamic /opt/pfring/lib/libpcap.so -Wl,-rpath,/opt/pfring/lib /usr/bin/ld: warning: libsnf.so.0, needed by /opt/pfring/lib/libpcap.so, not found (try using -rpath or -rpath-link) /opt/pfring/lib/libpcap.so: undefined reference to `clock_gettime' /opt/pfring/lib/libpcap.so: undefined reference to `snf_ring_open at snf_0.1' Uhm, looks like I have mixed in the snf == Myricom somehow in here. Looks like it's a bad idea to build pfring and myricom linked pcap applications on the same host. Sorry for the noise and thanks for the hint where to look for errors :-) On Thu, Feb 12, 2015 at 4:35 PM, Siwek, Jon wrote: > >> On Feb 12, 2015, at 9:14 AM, Micha? Purzy?ski wrote: >> >> When trying to build Bro 2.3.2 with libpcap from pf_ring 6.0.2 >> (./configure --with-pcap=/usr/local/lib, where libpcap-1.1.1-ring is >> installed) I get errors: >> >> -- Looking for include files HAVE_PCAP_INT_H >> -- Looking for include files HAVE_PCAP_INT_H - not found. >> -- Looking for pcap_freecode >> -- Looking for pcap_freecode - not found >> -- No implementation for pcap_freecode() >> -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_HAS_ERROR_PARAMETER >> -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_HAS_ERROR_PARAMETER - Failed >> -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_NO_ERROR_PARAMETER >> -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_NO_ERROR_PARAMETER - Failed >> CMake Error at cmake/PCAPTests.cmake:58 (message): >> Can't determine if pcap_compile_nopcap takes an error parameter >> Call Stack (most recent call first): >> CMakeLists.txt:176 (include) >> -- Configuring incomplete, errors occurred! > > Can you send the contents of build/CMakeFiles/CMakeError.log ? > > - Jon From michalpurzynski1 at gmail.com Thu Feb 12 08:31:47 2015 From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=) Date: Thu, 12 Feb 2015 17:31:47 +0100 Subject: [Bro] Bro 2.3.2 and pf_ring 6.0.2 In-Reply-To: <54DCD076.5020807@louruppert.com> References: <54DCD076.5020807@louruppert.com> Message-ID: Yep, confirmed LDFLAGS exported to "-Wl,--no-as-needed -lrt" is needed or the configure fails on Ubuntu with /opt/pfring/lib/libpcap.so: undefined reference to `clock_gettime' On Thu, Feb 12, 2015 at 5:10 PM, Lou Ruppert wrote: > Siwek, Jon: >> >>> On Feb 12, 2015, at 9:14 AM, Micha? Purzy?ski wrote: >>> >>> When trying to build Bro 2.3.2 with libpcap from pf_ring 6.0.2 >>> (./configure --with-pcap=/usr/local/lib, where libpcap-1.1.1-ring is >>> installed) I get errors: >>> >>> -- Looking for include files HAVE_PCAP_INT_H >>> -- Looking for include files HAVE_PCAP_INT_H - not found. >>> -- Looking for pcap_freecode >>> -- Looking for pcap_freecode - not found >>> -- No implementation for pcap_freecode() >>> -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_HAS_ERROR_PARAMETER >>> -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_HAS_ERROR_PARAMETER - Failed >>> -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_NO_ERROR_PARAMETER >>> -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_NO_ERROR_PARAMETER - Failed >>> CMake Error at cmake/PCAPTests.cmake:58 (message): >>> Can't determine if pcap_compile_nopcap takes an error parameter >>> Call Stack (most recent call first): >>> CMakeLists.txt:176 (include) >>> -- Configuring incomplete, errors occurred! >> >> Can you send the contents of build/CMakeFiles/CMakeError.log ? > > I wish I'd thought of that. Instead I kludged a quick solution: > > LDFLAGS="-Wl,--no-as-needed -lrt" ./configure > > I'm not sure why that bit of ugliness helps, but it got me up and > running again. > > -lou > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From edthoma at sandia.gov Thu Feb 12 09:50:02 2015 From: edthoma at sandia.gov (Thomas, Eric D) Date: Thu, 12 Feb 2015 17:50:02 +0000 Subject: [Bro] Include VLAN ID tag in output Message-ID: I know how one would add a new column to output files. What is the structure.member that contains the VLAN ID? -- Eric Thomas edthoma at sandia.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150212/e5276c23/attachment.html From seth at icir.org Thu Feb 12 17:06:01 2015 From: seth at icir.org (Seth Hall) Date: Thu, 12 Feb 2015 20:06:01 -0500 Subject: [Bro] Include VLAN ID tag in output In-Reply-To: References: Message-ID: > On Feb 12, 2015, at 12:50 PM, Thomas, Eric D wrote: > > I know how one would add a new column to output files. What is the structure.member that contains the VLAN ID? VLANs are not currently made available at the script layer. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From pearce at cs.berkeley.edu Tue Feb 17 16:20:18 2015 From: pearce at cs.berkeley.edu (Paul Pearce) Date: Tue, 17 Feb 2015 16:20:18 -0800 Subject: [Bro] Bro's escaping of non-printable characters behaves unexpected Message-ID: Hello everyone, I'm encountering a problem where I am unable to reconstruct original inputs from bro log files. This example summarizes the problem: ---- $ bro -e 'event bro_init() { print "foo\x00bar\\0baz"; }' foo\0bar\0baz ---- This makes recovering the original input impossible, as you can't differentiate between the escaped null and the ascii characters '\' and '0'. If bro was going to implicitly escape the string, I would have expected the following output: ---- $ bro -e 'event bro_init() { print "foo\x00bar\\0baz"; }' foo\0bar\\0baz ---- A workaround would be to output files in raw mode, however I am encountering this problem with logs generated via the logging framework, which supports no such option (AFAIK). Another workaround would be to substitute '\' for '\\' in all such outputs before handing them to the logging framework, but that solution seems... sub par. My read here is that bro's auto-escaping functionality should be changed to allow reconstruction of inputs in all cases. Thanks. -Paul From pearce at cs.berkeley.edu Tue Feb 17 17:15:31 2015 From: pearce at cs.berkeley.edu (Paul Pearce) Date: Tue, 17 Feb 2015 17:15:31 -0800 Subject: [Bro] Bro's escaping of non-printable characters behaves unexpected In-Reply-To: References: Message-ID: Hello, That was a poor example, as it used \0 which is special cased by the bro escape functionality. This problem also extends beyond non-printable to non-ascii (unicode) characters. Here's another example with a unicode character for the registered sign ? (\xc2\xae). ---- $ bro -e 'event bro_init() { print "foo \xc2\xae bar \\xc2\\xae baz"; }' foo \xc2\xae bar \xc2\xae baz ---- If you decide to revisit bro's escape functionality, I'd also point out that Bro special casing NUL, DEL, and ord(char) <= 26 creates difficulty when decoding bro output in other languages (e.g. python). Besides having different encodings based on the character range, the ^[A-Z] format causes the same ambiguous output issue above, but with ^ instead of \. Example: bro -e 'event bro_init() { print "foo \16 bar ^N baz"; }' If this is desired behavior, I might suggest an configuration option that allows ascii log generation using a standard representation for non-ascii/non-printable characters? On Tue, Feb 17, 2015 at 4:20 PM, Paul Pearce wrote: > Hello everyone, > > I'm encountering a problem where I am unable to reconstruct original > inputs from bro log files. This example summarizes the problem: > > ---- > $ bro -e 'event bro_init() { print "foo\x00bar\\0baz"; }' > > foo\0bar\0baz > ---- > > This makes recovering the original input impossible, as you can't > differentiate between the escaped null and the ascii characters '\' > and '0'. > > If bro was going to implicitly escape the string, I would have > expected the following output: > > ---- > $ bro -e 'event bro_init() { print "foo\x00bar\\0baz"; }' > > foo\0bar\\0baz > ---- > > A workaround would be to output files in raw mode, however I am > encountering this problem with logs generated via the logging > framework, which supports no such option (AFAIK). > > Another workaround would be to substitute '\' for '\\' in all such > outputs before handing them to the logging framework, but that > solution seems... sub par. > > My read here is that bro's auto-escaping functionality should be > changed to allow reconstruction of inputs in all cases. > > Thanks. > -Paul From johanna at icir.org Tue Feb 17 18:12:22 2015 From: johanna at icir.org (Johanna Amann) Date: Tue, 17 Feb 2015 18:12:22 -0800 Subject: [Bro] Bro's escaping of non-printable characters behaves unexpected In-Reply-To: References: Message-ID: <20150218021216.GA50916@Beezling.local> Hello Paul, I think the reason that the ascii writer of the logging framework of Bro does not support arbitrary binary data is, that it was conceived as a framework for writing human-readable log files, not arbitrary binary data. If you want to write binary data to log files, I would recommend just base64-encoding it before using the encode_base64 bif. If you are ok with just using the standard methods for writing to files outside of the logging framework, you can put them into binary mode, as you probably are aware. Johanna On Tue, Feb 17, 2015 at 05:15:31PM -0800, Paul Pearce wrote: > Hello, > > That was a poor example, as it used \0 which is special cased by the > bro escape functionality. > > This problem also extends beyond non-printable to non-ascii (unicode) > characters. Here's another example with a unicode character for the > registered sign ? (\xc2\xae). > > ---- > $ bro -e 'event bro_init() { print "foo \xc2\xae bar \\xc2\\xae baz"; }' > > foo \xc2\xae bar \xc2\xae baz > ---- > > If you decide to revisit bro's escape functionality, I'd also point > out that Bro special casing NUL, DEL, and ord(char) <= 26 creates > difficulty when decoding bro output in other languages (e.g. python). > > Besides having different encodings based on the character range, the > ^[A-Z] format causes the same ambiguous output issue above, but with ^ > instead of \. Example: bro -e 'event bro_init() { print "foo \16 > bar ^N baz"; }' > > If this is desired behavior, I might suggest an configuration option > that allows ascii log generation using a standard representation for > non-ascii/non-printable characters? > > On Tue, Feb 17, 2015 at 4:20 PM, Paul Pearce wrote: > > Hello everyone, > > > > I'm encountering a problem where I am unable to reconstruct original > > inputs from bro log files. This example summarizes the problem: > > > > ---- > > $ bro -e 'event bro_init() { print "foo\x00bar\\0baz"; }' > > > > foo\0bar\0baz > > ---- > > > > This makes recovering the original input impossible, as you can't > > differentiate between the escaped null and the ascii characters '\' > > and '0'. > > > > If bro was going to implicitly escape the string, I would have > > expected the following output: > > > > ---- > > $ bro -e 'event bro_init() { print "foo\x00bar\\0baz"; }' > > > > foo\0bar\\0baz > > ---- > > > > A workaround would be to output files in raw mode, however I am > > encountering this problem with logs generated via the logging > > framework, which supports no such option (AFAIK). > > > > Another workaround would be to substitute '\' for '\\' in all such > > outputs before handing them to the logging framework, but that > > solution seems... sub par. > > > > My read here is that bro's auto-escaping functionality should be > > changed to allow reconstruction of inputs in all cases. > > > > Thanks. > > -Paul > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > From pearce at cs.berkeley.edu Tue Feb 17 21:15:08 2015 From: pearce at cs.berkeley.edu (Paul Pearce) Date: Tue, 17 Feb 2015 21:15:08 -0800 Subject: [Bro] Bro's escaping of non-printable characters behaves unexpected In-Reply-To: <20150218021216.GA50916@Beezling.local> References: <20150218021216.GA50916@Beezling.local> Message-ID: Hey Johanna, Thanks for taking the time to respond. > I think the reason that the ascii writer of the logging framework of Bro > does not support arbitrary binary data is, that it was conceived as a > framework for writing human-readable log files, not arbitrary binary data. I'm going to push back a bit on characterizing this as supporting arbitrary binary data. These are unicode characters appearing in URIs ($http$URI) that I'm encountering in actual network traffic. I'm actually encountering them somewhat frequently. The problem manifests itself in the standard http.log, as well as the extensions I'm working on. I realize the RFC does not permit unicode in URLs, but given that they do occur in practice (browsers will just silently handle them), this seems like something worth supporting. I'll also point out that Bro's ascii logging facilities do currently support logging these characters, they simply do so in an unrecoverable/non-canonical way. What I'm proposing is standardization/cleanup for the escaping that Bro already performs. Thanks. -Paul From struck at ICSI.Berkeley.EDU Wed Feb 18 05:22:30 2015 From: struck at ICSI.Berkeley.EDU (Christian Struck) Date: Wed, 18 Feb 2015 14:22:30 +0100 Subject: [Bro] Bro's escaping of non-printable characters behaves unexpected In-Reply-To: References: <20150218021216.GA50916@Beezling.local> Message-ID: <54E49216.1040906@icsi.berkeley.edu> Hey Paul, On 18.02.2015 06:15, Paul Pearce wrote: > I realize the RFC does not permit unicode in URLs, but given that they > do occur in practice (browsers will just silently handle them), this > seems like something worth supporting. I think what you are looking for is this. http://en.wikipedia.org/wiki/Internationalized_resource_identifier > > Thanks. > -Paul Best Christian From michael.wenthold at gmail.com Thu Feb 19 08:33:11 2015 From: michael.wenthold at gmail.com (Michael Wenthold) Date: Thu, 19 Feb 2015 16:33:11 +0000 Subject: [Bro] Question about scan whitelisting ... Message-ID: I've been tinkering with the scan detection in Bro (2.3.2) and I was wondering if this was the most effective method for whitelisting hosts: const scanners_whitelist { x.x.x.x }; hook Notice::policy(n: Notice::Info) { if ( n$note == Scan::Port_Scan && n?$src && (n$src in scanners_whitelist) ) { print n$src; delete n$actions[Notice::ACTION_LOG]; }; } Please let me know if there's a better/more efficient method. Thanks! Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150219/fc367c9b/attachment.html From jdopheid at illinois.edu Thu Feb 19 10:30:43 2015 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Thu, 19 Feb 2015 18:30:43 +0000 Subject: [Bro] BroCon '15: Registration open Message-ID: Bro Community, BroCon '15 registration is now open. You may register here: https://www.regonline.com/brocon2015 We have reserved a block of hotel rooms for the event. For more information about hotel accommodations and other updates, see the event page: https://www.bro.org/community/brocon2015.html Thanks for your continued support, see you in August! Regards, The Bro Team ------ Jeannette M. Dopheide Bro Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From matt.clemons at gmail.com Mon Feb 23 14:32:52 2015 From: matt.clemons at gmail.com (Matt Clemons) Date: Mon, 23 Feb 2015 16:32:52 -0600 Subject: [Bro] Log Source Message-ID: Is there a way to add Worker source to all bro logs? I was able to do this with the conn.log, but if i try others, bad things happen. Can someone help? redef record Conn::Info += { peer_descr: string &default="unknown" &log; }; event connection_state_remove(c: connection){ c$conn$peer_descr = peer_description; } -- Regards, Matt Clemons -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150223/18ed158c/attachment.html From gfaulkner.nsm at gmail.com Tue Feb 24 10:23:59 2015 From: gfaulkner.nsm at gmail.com (Gary Faulkner) Date: Tue, 24 Feb 2015 12:23:59 -0600 Subject: [Bro] Bro 2.3.2-419 segfaults when using PF_RING 6.0.3 libpcap 1.6.2 and pfdnacluster_master on RHEL 6.6 Message-ID: <54ECC1BF.8070106@gmail.com> Hello, I?m having trouble getting Bro to run with PF_RING after updating from RHEL 6.5 to RHEL 6.6. The PF_RING aware drivers (DNA/ZC etc) in the ?stable? 6.0.2 branch of PF_RING don?t appear to compile correctly on RHEL 6.6, which necessitated a move to the latest 6.0.3 development branch (rev.9009). This version compiles fine and I have it working with both Suricata and nprobe, but can?t get it working with Bro. Bro doesn?t seem to be able to open the dnacluster:21 at 0 etc interfaces with the new version. Specifically bro segfaults when calling the PF_RING version of libpcap.so.1.6.2, which is a new version of libpcap in 6.0.3. Previously libpcap was 1.1.1. I have also tried to compile PF_RING 6.0.2 stable on RHEL 6.6 with the newer drivers, but the version of pfdnacluster_master that ships with PF_RING 6.0.2 stable (that uses the older libpcap) will silently crash on RHEL 6.6. I?ve attached the output of a broctl diag to this email. Typically when I've seen an error where bro can?t listen on dnacluster in the past it has been due to the interface already being in use, bro not being able to find pfring, or not compiling against the correct libpcap. I?ve verified this isn?t the case to the best of my ability (no other libpcap on the system, fresh dna driver load and instance of pfdnaclster_master, pfring in $PATH etc). I?ve also verified that I can see packets on the dnacluster interfaces by testing with pfcount. It looks like perhaps bro doesn?t like the new version of libpcap. I have tried compiling and running bro with debugging enabled, but bro seems to crash on the workers without generating anything in the various debug.log files. Any thoughts? Here are example error messages from /var/log/messages: kernel: bro[1653]: segfault at 1371670 ip 00007f5a9e7f0660 sp 00007fff8714b300 error 4 in libpcap.so.1.6.2[7f5a9e7d9000+90000] kernel: bro[1643]: segfault at 1371670 ip 00007ff16d19b660 sp 00007fff81eea9a0 error 4 in libpcap.so.1.6.2[7ff16d184000+90000] kernel: bro[1656]: segfault at 1371670 ip 00007fcf3c6cf660 sp 00007fff3e1789b0 error 4 in libpcap.so.1.6.2[7fcf3c6b8000+90000] kernel: bro[1644]: segfault at 1 ip 00007f5932268506 sp 00007fffcd3ea0b0 error 4 in libpcap.so.1.6.2[7f5932251000+90000] kernel: bro[1642]: segfault at 1 ip 00007ff3d1c83506 sp 00007fff468f4930 error 4 in libpcap.so.1.6.2[7ff3d1c6c000+90000] kernel: bro[1658]: segfault at 1371670 ip 00007f53584f2660 sp 00007ffff89515f0 error 4 in libpcap.so.1.6.2[7f53584db000+90000] kernel: bro[1652]: segfault at 1371670 ip 00007f158fbc7660 sp 00007fff14aa7e20 error 4 in libpcap.so.1.6.2[7f158fbb0000+90000] kernel: bro[1660]: segfault at 1371670 ip 00007f2fee8e7660 sp 00007ffff9dacaf0 error 4 in libpcap.so.1.6.2[7f2fee8d0000+90000] kernel: bro[1641]: segfault at 1 ip 00007f32fbc48506 sp 00007fff7d9b2a00 error 4 in libpcap.so.1.6.2[7f32fbc31000+90000] kernel: bro[1662]: segfault at b836210 ip 00007f5c9d669660 sp 00007fff71636fb0 error 4 in libpcap.so.1.6.2[7f5c9d652000+90000] kernel: bro[4220]: segfault at 1371670 ip 00007f6d35299660 sp 00007fff4d896940 error 4 in libpcap.so.1.6.2[7f6d35282000+90000] kernel: bro[4465]: segfault at 1371670 ip 00007f202ff75660 sp 00007fff04fff8c0 error 4 in libpcap.so.1.6.2[7f202ff5e000+90000] kernel: bro[4710]: segfault at 1371670 ip 00007fd8bc794660 sp 00007fff33041db0 error 4 in libpcap.so.1.6.2[7fd8bc77d000+90000] kernel: bro[7873]: segfault at 1371670 ip 00007ffc910f2660 sp 00007fff1b5ba1b0 error 4 in libpcap.so.1.6.2[7ffc910db000+90000] kernel: bro[8065]: segfault at 1371670 ip 00007ffaa5c8f660 sp 00007fff3cdde390 error 4 in libpcap.so.1.6.2[7ffaa5c78000+90000] kernel: bro[8257]: segfault at 63745e0 ip 00007ff913224660 sp 00007fff297ca2f0 error 4 in libpcap.so.1.6.2[7ff91320d000+90000] kernel: bro[8446]: segfault at 1371670 ip 00007f0a1c567660 sp 00007fffdf059910 error 4 in libpcap.so.1.6.2[7f0a1c550000+90000] kernel: bro[8638]: segfault at 1371670 ip 00007f50982af660 sp 00007fff703caa30 error 4 in libpcap.so.1.6.2[7f5098298000+90000] kernel: bro[8835]: segfault at 1371670 ip 00007f1b4acd2660 sp 00007fffacc16630 error 4 in libpcap.so.1.6.2[7f1b4acbb000+90000] kernel: bro[9036]: segfault at 1 ip 00007f10df91b506 sp 00007fff5ac3e320 error 4 in libpcap.so.1.6.2[7f10df904000+90000] Regards, Gary -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150224/8c3b09f6/attachment-0001.html -------------- next part -------------- [manager] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== reporter.log 0.000000 Reporter::ERROR count underflow (--Cluster::worker_count) /nsm/bro/share/bro/base/frameworks/cluster/./main.bro, line 150 1424727407.765899 Reporter::INFO processing continued (empty) 0.000000 Reporter::ERROR count underflow (--Cluster::worker_count) /nsm/bro/share/bro/base/frameworks/cluster/./main.bro, line 150 0.000000 Reporter::ERROR count underflow (--Cluster::worker_count) /nsm/bro/share/bro/base/frameworks/cluster/./main.bro, line 150 1424727412.126406 Reporter::INFO processing continued (empty) 1424727414.338706 Reporter::INFO processing suspended (empty) 1424727414.338727 Reporter::INFO processing continued (empty) 0.000000 Reporter::ERROR count underflow (--Cluster::worker_count) /nsm/bro/share/bro/base/frameworks/cluster/./main.bro, line 150 0.000000 Reporter::ERROR count underflow (--Cluster::worker_count) /nsm/bro/share/bro/base/frameworks/cluster/./main.bro, line 150 0.000000 Reporter::ERROR count underflow (--Cluster::worker_count) /nsm/bro/share/bro/base/frameworks/cluster/./main.bro, line 150 ==== stderr.log ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl base/frameworks/cluster local-manager.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/nsm/bin:/nsm/share:/nsm/man:/home/nsm/bin:/nsm/bro:/nsm/bro/bin:/nsm/pfring/bin:/nsm/pfring/sbin:/nsm/pfring/include:/nsm/pfring/include/linux:/nsm/pfring/lib:/nsm/pfring/modules:/nsm/pfring/share:/nsm/PF_RING/userland/lib:/usr/share/GeoIP:/usr/lib64:/usr/include:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=manager ==== .status RUNNING [net_run] ==== prof.log 1424727431.270431 known_services/Log::WRITER_ASCII in=173 out=139 pending=0/0 (#queue r/w: in=173/173 out=139/139) 1424727431.270431 x509/Log::WRITER_ASCII in=174 out=139 pending=0/0 (#queue r/w: in=174/174 out=139/139) 1424727431.270431 ssl/Log::WRITER_ASCII in=174 out=139 pending=0/0 (#queue r/w: in=174/174 out=139/139) 1424727431.270431 notice/Log::WRITER_ASCII in=175 out=139 pending=0/0 (#queue r/w: in=175/175 out=139/139) 1424727431.270431 syslog/Log::WRITER_ASCII in=150 out=139 pending=0/0 (#queue r/w: in=150/150 out=139/139) 1424727431.270431 known_certs/Log::WRITER_ASCII in=159 out=139 pending=0/0 (#queue r/w: in=159/159 out=139/139) 1424727431.270431 ftp/Log::WRITER_ASCII in=144 out=139 pending=0/0 (#queue r/w: in=144/144 out=139/139) 1424727431.270431 dpd/Log::WRITER_ASCII in=152 out=139 pending=0/0 (#queue r/w: in=152/152 out=139/139) 1424727431.270431 conn/Log::WRITER_ASCII in=150 out=138 pending=0/0 (#queue r/w: in=150/150 out=138/138) 1424727431.270431 smtp/Log::WRITER_ASCII in=139 out=136 pending=0/0 (#queue r/w: in=139/139 out=136/136) ==== packet_filter.log #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path packet_filter #open 2015-02-23-15-34-51 #fields ts node filter init success #types time string string bool bool [worker-1-1] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log listening on dnacluster:21 at 0, capture length 8192 bytes 1424727385.842901 processing suspended 1424727385.842938 processing continued /nsm/bro/share/broctl/scripts/run-bro: line 85: 7873 Segmentation fault nohup $mybro "$@" ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 0 -U .status -p broctl -p broctl-live -p local -p worker-1-1 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-1 ==== .status RUNNING [net_run] ==== prof.log 1424727385.837440 TCP-States:Rst. 1424727385.837440 Connections expired due to inactivity: 0 1424727385.837440 Total reassembler data: 0K 1424727385.837440 Timers: current=37 max=38 mem=2K lag=1424727384.84s 1424727385.837440 DNS_Mgr: requests=0 succesful=0 failed=0 pending=0 cached_hosts=0 cached_addrs=0 1424727385.837440 Triggers: total=0 pending=0 1424727385.837440 RotateTimer = 3 1424727385.837440 ScheduleTimer = 12 1424727385.837440 TableValTimer = 22 1424727385.837440 Threads: current=0 ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-10] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log listening on dnacluster:21 at 1, capture length 8192 bytes 1424727388.066983 processing suspended 1424727388.067015 processing continued /nsm/bro/share/broctl/scripts/run-bro: line 85: 8065 Segmentation fault nohup $mybro "$@" ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 1 -U .status -p broctl -p broctl-live -p local -p worker-1-10 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-10 ==== .status RUNNING [net_run] ==== prof.log 1424727388.060710 TCP-States:Rst. 1424727388.060710 Connections expired due to inactivity: 0 1424727388.060710 Total reassembler data: 0K 1424727388.060710 Timers: current=37 max=38 mem=2K lag=1424727387.06s 1424727388.060710 DNS_Mgr: requests=0 succesful=0 failed=0 pending=0 cached_hosts=0 cached_addrs=0 1424727388.060710 Triggers: total=0 pending=0 1424727388.060710 RotateTimer = 3 1424727388.060710 ScheduleTimer = 12 1424727388.060710 TableValTimer = 22 1424727388.060710 Threads: current=0 ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-11] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log listening on dnacluster:21 at 2, capture length 8192 bytes 1424727390.295242 processing suspended 1424727390.295280 processing continued /nsm/bro/share/broctl/scripts/run-bro: line 85: 8257 Segmentation fault nohup $mybro "$@" ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 2 -U .status -p broctl -p broctl-live -p local -p worker-1-11 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-11 ==== .status RUNNING [net_run] ==== prof.log 1424727390.288829 TCP-States:Rst. 1424727390.288829 Connections expired due to inactivity: 0 1424727390.288829 Total reassembler data: 0K 1424727390.288829 Timers: current=37 max=38 mem=2K lag=1424727389.29s 1424727390.288829 DNS_Mgr: requests=0 succesful=0 failed=0 pending=0 cached_hosts=0 cached_addrs=0 1424727390.288829 Triggers: total=0 pending=0 1424727390.288829 RotateTimer = 3 1424727390.288829 ScheduleTimer = 12 1424727390.288829 TableValTimer = 22 1424727390.288829 Threads: current=0 ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-12] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log listening on dnacluster:21 at 3, capture length 8192 bytes 1424727392.477571 processing suspended 1424727392.477593 processing continued /nsm/bro/share/broctl/scripts/run-bro: line 85: 8446 Segmentation fault nohup $mybro "$@" ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 3 -U .status -p broctl -p broctl-live -p local -p worker-1-12 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-12 ==== .status RUNNING [net_run] ==== prof.log 1424727392.471928 TCP-States:Rst. 1424727392.471928 Connections expired due to inactivity: 0 1424727392.471928 Total reassembler data: 0K 1424727392.471928 Timers: current=37 max=38 mem=2K lag=1424727391.47s 1424727392.471928 DNS_Mgr: requests=0 succesful=0 failed=0 pending=0 cached_hosts=0 cached_addrs=0 1424727392.471928 Triggers: total=0 pending=0 1424727392.471928 RotateTimer = 3 1424727392.471928 ScheduleTimer = 12 1424727392.471928 TableValTimer = 22 1424727392.471928 Threads: current=0 ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-13] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log listening on dnacluster:21 at 4, capture length 8192 bytes 1424727394.641198 processing suspended 1424727394.641244 processing continued /nsm/bro/share/broctl/scripts/run-bro: line 85: 8638 Segmentation fault nohup $mybro "$@" ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 4 -U .status -p broctl -p broctl-live -p local -p worker-1-13 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-13 ==== .status RUNNING [net_run] ==== prof.log 1424727394.634207 TCP-States:Rst. 1424727394.634207 Connections expired due to inactivity: 0 1424727394.634207 Total reassembler data: 0K 1424727394.634207 Timers: current=37 max=38 mem=2K lag=1424727393.63s 1424727394.634207 DNS_Mgr: requests=0 succesful=0 failed=0 pending=0 cached_hosts=0 cached_addrs=0 1424727394.634207 Triggers: total=0 pending=0 1424727394.634207 RotateTimer = 3 1424727394.634207 ScheduleTimer = 12 1424727394.634207 TableValTimer = 22 1424727394.634207 Threads: current=0 ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-14] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log listening on dnacluster:21 at 5, capture length 8192 bytes 1424727396.855224 processing suspended 1424727396.855269 processing continued /nsm/bro/share/broctl/scripts/run-bro: line 85: 8835 Segmentation fault nohup $mybro "$@" ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 5 -U .status -p broctl -p broctl-live -p local -p worker-1-14 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-14 ==== .status RUNNING [net_run] ==== prof.log 1424727396.849169 TCP-States:Rst. 1424727396.849169 Connections expired due to inactivity: 0 1424727396.849169 Total reassembler data: 0K 1424727396.849169 Timers: current=37 max=38 mem=2K lag=1424727395.85s 1424727396.849169 DNS_Mgr: requests=0 succesful=0 failed=0 pending=0 cached_hosts=0 cached_addrs=0 1424727396.849169 Triggers: total=0 pending=0 1424727396.849169 RotateTimer = 3 1424727396.849169 ScheduleTimer = 12 1424727396.849169 TableValTimer = 22 1424727396.849169 Threads: current=0 ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-15] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log listening on dnacluster:21 at 6, capture length 8192 bytes 1424727315.591060 processing suspended 1424727315.591087 processing continued /nsm/bro/share/broctl/scripts/run-bro: line 85: 4220 Segmentation fault nohup $mybro "$@" ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 6 -U .status -p broctl -p broctl-live -p local -p worker-1-15 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-15 ==== .status RUNNING [net_run] ==== prof.log 1424727315.584755 TCP-States:Rst. 1424727315.584755 Connections expired due to inactivity: 0 1424727315.584755 Total reassembler data: 0K 1424727315.584755 Timers: current=37 max=38 mem=2K lag=1424727314.58s 1424727315.584755 DNS_Mgr: requests=0 succesful=0 failed=0 pending=0 cached_hosts=0 cached_addrs=0 1424727315.584755 Triggers: total=0 pending=0 1424727315.584755 RotateTimer = 3 1424727315.584755 ScheduleTimer = 12 1424727315.584755 TableValTimer = 22 1424727315.584755 Threads: current=0 ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-16] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log listening on dnacluster:21 at 7, capture length 8192 bytes 1424727399.037673 processing suspended 1424727399.037757 processing continued /nsm/bro/share/broctl/scripts/run-bro: line 85: 9036 Segmentation fault nohup $mybro "$@" ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 7 -U .status -p broctl -p broctl-live -p local -p worker-1-16 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-16 ==== .status RUNNING [net_run] ==== prof.log 1424727399.029738 TCP-States:Rst. 1424727399.029738 Connections expired due to inactivity: 0 1424727399.029738 Total reassembler data: 0K 1424727399.029738 Timers: current=37 max=38 mem=2K lag=1424727398.03s 1424727399.029738 DNS_Mgr: requests=0 succesful=0 failed=0 pending=0 cached_hosts=0 cached_addrs=0 1424727399.029738 Triggers: total=0 pending=0 1424727399.029738 RotateTimer = 3 1424727399.029738 ScheduleTimer = 12 1424727399.029738 TableValTimer = 22 1424727399.029738 Threads: current=0 ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-17] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log listening on dnacluster:21 at 8, capture length 8192 bytes 1424727318.002349 processing suspended 1424727318.002371 processing continued /nsm/bro/share/broctl/scripts/run-bro: line 85: 4465 Segmentation fault nohup $mybro "$@" ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 8 -U .status -p broctl -p broctl-live -p local -p worker-1-17 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-17 ==== .status RUNNING [net_run] ==== prof.log 1424727317.996649 TCP-States:Rst. 1424727317.996649 Connections expired due to inactivity: 0 1424727317.996649 Total reassembler data: 0K 1424727317.996649 Timers: current=37 max=38 mem=2K lag=1424727317.00s 1424727317.996649 DNS_Mgr: requests=0 succesful=0 failed=0 pending=0 cached_hosts=0 cached_addrs=0 1424727317.996649 Triggers: total=0 pending=0 1424727317.996649 RotateTimer = 3 1424727317.996649 ScheduleTimer = 12 1424727317.996649 TableValTimer = 22 1424727317.996649 Threads: current=0 ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-18] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log listening on dnacluster:21 at 9, capture length 8192 bytes 1424727320.433816 processing suspended 1424727320.433889 processing continued /nsm/bro/share/broctl/scripts/run-bro: line 85: 4710 Segmentation fault nohup $mybro "$@" ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 9 -U .status -p broctl -p broctl-live -p local -p worker-1-18 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-18 ==== .status RUNNING [net_run] ==== prof.log 1424727320.384226 TCP-States:Rst. 1424727320.384226 Connections expired due to inactivity: 0 1424727320.384226 Total reassembler data: 0K 1424727320.384226 Timers: current=37 max=38 mem=2K lag=1424727319.38s 1424727320.384226 DNS_Mgr: requests=0 succesful=0 failed=0 pending=0 cached_hosts=0 cached_addrs=0 1424727320.384226 Triggers: total=0 pending=0 1424727320.384226 RotateTimer = 3 1424727320.384226 ScheduleTimer = 12 1424727320.384226 TableValTimer = 22 1424727320.384226 Threads: current=0 ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-19] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log fatal error: problem with interface dnacluster:21 at 10 (dnacluster:21 at 10: No such device exists (No such device exists)) ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 10 -U .status -p broctl -p broctl-live -p local -p worker-1-19 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-19 ==== .status TERMINATED [atexit] ==== prof.log ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-2] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log fatal error: problem with interface dnacluster:21 at 11 (dnacluster:21 at 11: No such device exists (No such device exists)) ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 11 -U .status -p broctl -p broctl-live -p local -p worker-1-2 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-2 ==== .status TERMINATED [atexit] ==== prof.log ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-20] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log fatal error: problem with interface dnacluster:21 at 12 (dnacluster:21 at 12: No such device exists (No such device exists)) ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 12 -U .status -p broctl -p broctl-live -p local -p worker-1-20 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-20 ==== .status TERMINATED [atexit] ==== prof.log ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-21] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log fatal error: problem with interface dnacluster:21 at 13 (dnacluster:21 at 13: No such device exists (No such device exists)) ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 13 -U .status -p broctl -p broctl-live -p local -p worker-1-21 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-21 ==== .status TERMINATED [atexit] ==== prof.log ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-22] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log fatal error: problem with interface dnacluster:21 at 14 (dnacluster:21 at 14: No such device exists (No such device exists)) ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 14 -U .status -p broctl -p broctl-live -p local -p worker-1-22 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-22 ==== .status TERMINATED [atexit] ==== prof.log ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-3] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log fatal error: problem with interface dnacluster:21 at 15 (dnacluster:21 at 15: No such device exists (No such device exists)) ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 15 -U .status -p broctl -p broctl-live -p local -p worker-1-3 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-3 ==== .status TERMINATED [atexit] ==== prof.log ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-4] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log fatal error: problem with interface dnacluster:21 at 16 (dnacluster:21 at 16: No such device exists (No such device exists)) ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 16 -U .status -p broctl -p broctl-live -p local -p worker-1-4 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-4 ==== .status TERMINATED [atexit] ==== prof.log ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-5] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log fatal error: problem with interface dnacluster:21 at 17 (dnacluster:21 at 17: No such device exists (No such device exists)) ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 17 -U .status -p broctl -p broctl-live -p local -p worker-1-5 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-5 ==== .status TERMINATED [atexit] ==== prof.log ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-6] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log fatal error: problem with interface dnacluster:21 at 18 (dnacluster:21 at 18: No such device exists (No such device exists)) ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 18 -U .status -p broctl -p broctl-live -p local -p worker-1-6 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-6 ==== .status TERMINATED [atexit] ==== prof.log ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-7] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log fatal error: problem with interface dnacluster:21 at 19 (dnacluster:21 at 19: No such device exists (No such device exists)) ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 19 -U .status -p broctl -p broctl-live -p local -p worker-1-7 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-7 ==== .status TERMINATED [atexit] ==== prof.log ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-8] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log fatal error: problem with interface dnacluster:21 at 20 (dnacluster:21 at 20: No such device exists (No such device exists)) ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 20 -U .status -p broctl -p broctl-live -p local -p worker-1-8 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-8 ==== .status TERMINATED [atexit] ==== prof.log ==== No packet_filter.log ==== No loaded_scripts.log [worker-1-9] Bro 2.3-419 Linux 2.6.32-504.8.1.el6.x86_64 ==== No reporter.log ==== stderr.log fatal error: problem with interface dnacluster:21 at 21 (dnacluster:21 at 21: No such device exists (No such device exists)) ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dnacluster:21 at 21 -U .status -p broctl -p broctl-live -p local -p worker-1-9 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/nsm/bro/bin:/nsm/bro/share/broctl/scripts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/nsm/bro/share/bro:/nsm/bro/share/bro/policy:/nsm/bro/share/bro/site CLUSTER_NODE=worker-1-9 ==== .status TERMINATED [atexit] ==== prof.log ==== No packet_filter.log ==== No loaded_scripts.log From gfaulkner.nsm at gmail.com Tue Feb 24 12:59:42 2015 From: gfaulkner.nsm at gmail.com (Gary Faulkner) Date: Tue, 24 Feb 2015 14:59:42 -0600 Subject: [Bro] Bro 2.3.2-419 segfaults when using PF_RING 6.0.3 libpcap 1.6.2 and pfdnacluster_master on RHEL 6.6 In-Reply-To: References: <54ECC1BF.8070106@gmail.com> Message-ID: <54ECE63E.201@gmail.com> A couple folks have suggested I run this with gdb and get a backtrace to post here. Here is a quick gdb session with a backtrace of when I run bro -i dnacluster:21 at 0: # gdb /nsm/bro/bin/bro GNU gdb (GDB) SLES Expanded Support platform (7.2-75.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux". For bug reporting instructions, please see: ... Reading symbols from /nsm/bro/bin/bro...done. (gdb) run -i dnacluster:21 at 0 Starting program: /nsm/bro/bin/bro -i dnacluster:21 at 0 [Thread debugging using libthread_db enabled] listening on dnacluster:21 at 0, capture length 8192 bytes [New Thread 0x7fff20fd0700 (LWP 36513)] [New Thread 0x7fff1bfff700 (LWP 36514)] [New Thread 0x7fff1b5fe700 (LWP 36515)] [New Thread 0x7fff1abfd700 (LWP 36516)] [New Thread 0x7fff1a1fc700 (LWP 36517)] [New Thread 0x7fff197fb700 (LWP 36518)] [New Thread 0x7fff18dfa700 (LWP 36519)] [New Thread 0x7fff03fff700 (LWP 36520)] [New Thread 0x7fff035fe700 (LWP 36521)] [New Thread 0x7fff02bfd700 (LWP 36522)] [New Thread 0x7fff021fc700 (LWP 36523)] [New Thread 0x7fff017fb700 (LWP 36524)] Program received signal SIGSEGV, Segmentation fault. 0x00007ffff7959506 in pcap_read_packet (handle=0x2631640, callback=0x7ffff795d720 , userdata=0x7fffffffda20 "p\025c\002") at ./pcap-linux.c:1807 1807 ./pcap-linux.c: No such file or directory. in ./pcap-linux.c Missing separate debuginfos, use: debuginfo-install GeoIP-1.5.1-5.el6.x86_64 glibc-2.12-1.149.el6_6.5.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-33.el6.x86_64 libcom_err-1.41.12-21.el6.x86_64 libgcc-4.4.7-11.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libstdc++-4.4.7-11.el6.x86_64 numactl-2.0.9-2.el6.x86_64 openssl-1.0.1e-30.el6_6.5.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007ffff7959506 in pcap_read_packet (handle=0x2631640, callback=0x7ffff795d720 , userdata=0x7fffffffda20 "p\025c\002") at ./pcap-linux.c:1807 #1 0x00007ffff795d79b in pcap_next (p=, h=) at ./pcap.c:218 #2 0x0000000000a4a490 in iosource::pcap::PcapSource::ExtractNextPacket (this=0x2631430, pkt=0x2631468) at/nsm/bro/git/bro2.3-419/bro/ src/iosource/pcap/Source.cc:151 #3 0x0000000000a7580c in iosource::PktSrc::ExtractNextPacketInternal (this=0x2631430) at /nsm/bro/git/bro2.3-419/bro/src/iosource/PktSrc.cc:432 #4 0x0000000000a7511b in iosource::PktSrc::NextTimestamp (this=0x2631430, local_network_time=0x7fffffffdcb8) at /nsm/bro/git/bro2.3-419/bro/src/iosource/PktSrc.cc:241 #5 0x0000000000a71193 in iosource::Manager::FindSoonest (this=0xf29bc0, ts=0x7fffffffddc8) at/nsm/bro/git/bro2.3-419/bro/ src/iosource/Manager.cc:82 #6 0x00000000007895d1 in net_run () at/nsm/bro/git/bro2.3-419/bro/ src/Net.cc:301 #7 0x00000000006d8ed7 in main (argc=3, argv=0x7fffffffe498) at /nsm/bro/git/bro2.3-419/bro/src/main.cc:1200 On 2/24/2015 1:20 PM, John Donnelly wrote: > Can you use gdb to get a backstrace ? > > ... > > ---------- Forwarded message ---------- > From: Gary Faulkner > Date: Tue, Feb 24, 2015 at 12:23 PM > Subject: [Bro] Bro 2.3.2-419 segfaults when using PF_RING 6.0.3 libpcap > 1.6.2 and pfdnacluster_master on RHEL 6.6 > To: "bro at bro.org List" > > > Hello, > > I?m having trouble getting Bro to run with PF_RING after updating from RHEL > 6.5 to RHEL 6.6. The PF_RING aware drivers (DNA/ZC etc) in the ?stable? > 6.0.2 branch of PF_RING don?t appear to compile correctly on RHEL 6.6, > which necessitated a move to the latest 6.0.3 development branch > (rev.9009). This version compiles fine and I have it working with both > Suricata and nprobe, but can?t get it working with Bro. Bro doesn?t seem to > be able to open the dnacluster:21 at 0 etc interfaces with the new version. > Specifically bro segfaults when calling the PF_RING version of > libpcap.so.1.6.2, which is a new version of libpcap in 6.0.3. Previously > libpcap was 1.1.1. I have also tried to compile PF_RING 6.0.2 stable on > RHEL 6.6 with the newer drivers, but the version of pfdnacluster_master > that ships with PF_RING 6.0.2 stable (that uses the older libpcap) will > silently crash on RHEL 6.6. I?ve attached the output of a broctl diag to > this email. Typically when I've seen an error where bro can?t listen on > dnacluster in the past it has been due to the interface already being in > use, bro not being able to find pfring, or not compiling against the > correct libpcap. I?ve verified this isn?t the case to the best of my > ability (no other libpcap on the system, fresh dna driver load and instance > of pfdnaclster_master, pfring in $PATH etc). I?ve also verified that I can > see packets on the dnacluster interfaces by testing with pfcount. It looks > like perhaps bro doesn?t like the new version of libpcap. I have tried > compiling and running bro with debugging enabled, but bro seems to crash on > the workers without generating anything in the various debug.log files. Any > thoughts? > > Here are example error messages from /var/log/messages: > > kernel: bro[1653]: segfault at 1371670 ip 00007f5a9e7f0660 sp > 00007fff8714b300 error 4 in libpcap.so.1.6.2[7f5a9e7d9000+90000] > kernel: bro[1643]: segfault at 1371670 ip 00007ff16d19b660 sp > 00007fff81eea9a0 error 4 in libpcap.so.1.6.2[7ff16d184000+90000] > kernel: bro[1656]: segfault at 1371670 ip 00007fcf3c6cf660 sp > 00007fff3e1789b0 error 4 in libpcap.so.1.6.2[7fcf3c6b8000+90000] > kernel: bro[1644]: segfault at 1 ip 00007f5932268506 sp 00007fffcd3ea0b0 > error 4 in libpcap.so.1.6.2[7f5932251000+90000] > kernel: bro[1642]: segfault at 1 ip 00007ff3d1c83506 sp 00007fff468f4930 > error 4 in libpcap.so.1.6.2[7ff3d1c6c000+90000] > kernel: bro[1658]: segfault at 1371670 ip 00007f53584f2660 sp > 00007ffff89515f0 error 4 in libpcap.so.1.6.2[7f53584db000+90000] > kernel: bro[1652]: segfault at 1371670 ip 00007f158fbc7660 sp > 00007fff14aa7e20 error 4 in libpcap.so.1.6.2[7f158fbb0000+90000] > kernel: bro[1660]: segfault at 1371670 ip 00007f2fee8e7660 sp > 00007ffff9dacaf0 error 4 in libpcap.so.1.6.2[7f2fee8d0000+90000] > kernel: bro[1641]: segfault at 1 ip 00007f32fbc48506 sp 00007fff7d9b2a00 > error 4 in libpcap.so.1.6.2[7f32fbc31000+90000] > kernel: bro[1662]: segfault at b836210 ip 00007f5c9d669660 sp > 00007fff71636fb0 error 4 in libpcap.so.1.6.2[7f5c9d652000+90000] > kernel: bro[4220]: segfault at 1371670 ip 00007f6d35299660 sp > 00007fff4d896940 error 4 in libpcap.so.1.6.2[7f6d35282000+90000] > kernel: bro[4465]: segfault at 1371670 ip 00007f202ff75660 sp > 00007fff04fff8c0 error 4 in libpcap.so.1.6.2[7f202ff5e000+90000] > kernel: bro[4710]: segfault at 1371670 ip 00007fd8bc794660 sp > 00007fff33041db0 error 4 in libpcap.so.1.6.2[7fd8bc77d000+90000] > kernel: bro[7873]: segfault at 1371670 ip 00007ffc910f2660 sp > 00007fff1b5ba1b0 error 4 in libpcap.so.1.6.2[7ffc910db000+90000] > kernel: bro[8065]: segfault at 1371670 ip 00007ffaa5c8f660 sp > 00007fff3cdde390 error 4 in libpcap.so.1.6.2[7ffaa5c78000+90000] > kernel: bro[8257]: segfault at 63745e0 ip 00007ff913224660 sp > 00007fff297ca2f0 error 4 in libpcap.so.1.6.2[7ff91320d000+90000] > kernel: bro[8446]: segfault at 1371670 ip 00007f0a1c567660 sp > 00007fffdf059910 error 4 in libpcap.so.1.6.2[7f0a1c550000+90000] > kernel: bro[8638]: segfault at 1371670 ip 00007f50982af660 sp > 00007fff703caa30 error 4 in libpcap.so.1.6.2[7f5098298000+90000] > kernel: bro[8835]: segfault at 1371670 ip 00007f1b4acd2660 sp > 00007fffacc16630 error 4 in libpcap.so.1.6.2[7f1b4acbb000+90000] > kernel: bro[9036]: segfault at 1 ip 00007f10df91b506 sp 00007fff5ac3e320 > error 4 in libpcap.so.1.6.2[7f10df904000+90000] > > Regards, > Gary > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150224/736aa19a/attachment.html From td66bshwu at gmail.com Tue Feb 24 22:47:00 2015 From: td66bshwu at gmail.com (Lachlan Kang) Date: Wed, 25 Feb 2015 17:17:00 +1030 Subject: [Bro] Meaning of notices in weird.log Message-ID: Is there some kind of explanation page that describes the meaning of all the different notifications that can be found in weird.log? Specifically I want to learn what SYN_seq_jump means. Thanks. From gfaulkner.nsm at gmail.com Wed Feb 25 08:56:52 2015 From: gfaulkner.nsm at gmail.com (Gary Faulkner) Date: Wed, 25 Feb 2015 10:56:52 -0600 Subject: [Bro] Building Bro and have a couple errors when running configure Message-ID: <54EDFED4.4070505@gmail.com> Bro seems to complete the configure step, but I'm seeing a couple 'not found' and 'Failed' messages during when tests are run in the configure step and am wondering if these are errors that can be ignored or if I need to fix them first. This is on RHEL 6.6. I'm using the PF_RING libpcap, and the system libpcap is not installed. -- Performing Test ns_initparse_works_none - Failed -- Performing Test res_mkquery_works_none - Failed -- Looking for htonll - not found -- Looking for include file sys/ethernet.h - not found -- Looking for include file net/ethertypes.h - not found -- Looking for include file os-proto.h - not found -- Performing Test HAVE_READLINE_HISTORY_ENTRIES - Failed -- Performing Test SIN_LEN - Failed -- Looking for IPPROTO_IPV4 - not found -- Performing Test DO_SOCK_DECL - Failed -- Performing Test SYSLOG_INT - Failed -- Looking for include file pcap-int.h - not found -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_HAS_ERROR_PARAMETER - Failed I ended up installing readline-devel (no readline.h or history.h) and having to reinstall BIND (missing libresolv.a) at some point to resolve a few other not found issues. I'm not sure why the build script isn't finding pcap-int.h. I've manually added the pfring location of this file to my path and library path, but it doesn't seem to be finding the header file. I don't recall seeing some of these issues in previous builds, so I'm wondering if there are some new dependencies or perhaps if a recent update from RHEL 6.5 to 6.6 resulted in some weird issues. From jsiwek at illinois.edu Wed Feb 25 10:04:51 2015 From: jsiwek at illinois.edu (Siwek, Jon) Date: Wed, 25 Feb 2015 18:04:51 +0000 Subject: [Bro] Building Bro and have a couple errors when running configure In-Reply-To: <54EDFED4.4070505@gmail.com> References: <54EDFED4.4070505@gmail.com> Message-ID: <8AF6316C-153C-40A4-BAB6-1D23811BBE54@illinois.edu> > On Feb 25, 2015, at 10:56 AM, Gary Faulkner wrote: > > Bro seems to complete the configure step, but I'm seeing a couple 'not > found' and 'Failed' messages during when tests are run in the configure > step and am wondering if these are errors that can be ignored or if I > need to fix them first. This is on RHEL 6.6. I'm using the PF_RING > libpcap, and the system libpcap is not installed. > > -- Performing Test ns_initparse_works_none - Failed > -- Performing Test res_mkquery_works_none - Failed > -- Looking for htonll - not found > -- Looking for include file sys/ethernet.h - not found > -- Looking for include file net/ethertypes.h - not found > -- Looking for include file os-proto.h - not found > -- Performing Test HAVE_READLINE_HISTORY_ENTRIES - Failed > -- Performing Test SIN_LEN - Failed > -- Looking for IPPROTO_IPV4 - not found > -- Performing Test DO_SOCK_DECL - Failed > -- Performing Test SYSLOG_INT - Failed > -- Looking for include file pcap-int.h - not found > -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_HAS_ERROR_PARAMETER - Failed Individual tests that fail are usually fine as long as the overall configure/cmake script completes. The reason for these configure-time tests is because Bro has to work on various platforms and with differing versions of some libraries, so it has to check what is actually available to use and adjust accordingly. If the configure step completes, take that to mean ?no problems detected so far?. - Jon From liburdi.joshua at gmail.com Wed Feb 25 10:19:40 2015 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Wed, 25 Feb 2015 10:19:40 -0800 Subject: [Bro] Meaning of notices in weird.log In-Reply-To: References: Message-ID: I haven't seen a page describing what these mean, but the Bro project on github is useful for finding this type of info. If you take any of the strings listed in the actions table in the file scripts/base/frameworks/notice/weird.bro and search for them on github, you'll find where the weird is generated from. That should put you on the right track to figuring out what it means. https://github.com/bro/bro/search?utf8=%E2%9C%93&q=SYN_seq_jump&type=Code Josh On Tue, Feb 24, 2015 at 10:47 PM, Lachlan Kang wrote: > Is there some kind of explanation page that describes the meaning of > all the different notifications that can be found in weird.log? > Specifically I want to learn what SYN_seq_jump means. > > Thanks. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From raj at bivio.net Wed Feb 25 12:19:10 2015 From: raj at bivio.net (Raj Srinivasan) Date: Wed, 25 Feb 2015 20:19:10 +0000 Subject: [Bro] Question log/spool directory specification in broctl.conf Message-ID: I am running multiple bro workers in the same CPU, and would like to create different log and spool directories for each worker to avoid conflicts. Is there a worker specific meta variable I could use in the broctl.cfg file so that when broctl creates log and spool directories, it uses different file names for different workers? Alternatively, is there a way for all of the workers to use the same log and spool directories without conflicts? Currently, if I do that, I see workers crashing. Thanks, Raj -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150225/02403f64/attachment.html From dnthayer at illinois.edu Wed Feb 25 18:19:52 2015 From: dnthayer at illinois.edu (Daniel Thayer) Date: Wed, 25 Feb 2015 20:19:52 -0600 Subject: [Bro] Question log/spool directory specification in broctl.conf In-Reply-To: References: Message-ID: <54EE82C8.9010106@illinois.edu> Broctl automatically creates a separate working directory (such as "spool/worker-1") for each Bro process. All of the logs are automatically sent to the manager. On 02/25/2015 02:19 PM, Raj Srinivasan wrote: > I am running multiple bro workers in the same CPU, and would like to > create different log and spool directories for each worker to avoid > conflicts. Is there a worker specific meta variable I could use in the > broctl.cfg file so that when broctl creates log and spool directories, > it uses different file names for different workers? > > Alternatively, is there a way for all of the workers to use the same log > and spool directories without conflicts? Currently, if I do that, I see > workers crashing. > > Thanks, > > Raj From raj at bivio.net Wed Feb 25 21:33:21 2015 From: raj at bivio.net (Raj Srinivasan) Date: Thu, 26 Feb 2015 05:33:21 +0000 Subject: [Bro] Question log/spool directory specification in broctl.conf In-Reply-To: <54EE82C8.9010106@illinois.edu> References: <54EE82C8.9010106@illinois.edu> Message-ID: Thank you, Daniel. I did see the directories for different workers, so my question was more or less a shot in the dark. I will look some more into the worker crashes I have seen, and get back if I need any help. Thanks again! Raj -----Original Message----- From: Daniel Thayer [mailto:dnthayer at illinois.edu] Sent: Wednesday, February 25, 2015 6:20 PM To: Raj Srinivasan; bro at bro.org Subject: Re: [Bro] Question log/spool directory specification in broctl.conf Broctl automatically creates a separate working directory (such as "spool/worker-1") for each Bro process. All of the logs are automatically sent to the manager. On 02/25/2015 02:19 PM, Raj Srinivasan wrote: > I am running multiple bro workers in the same CPU, and would like to > create different log and spool directories for each worker to avoid > conflicts. Is there a worker specific meta variable I could use in the > broctl.cfg file so that when broctl creates log and spool directories, > it uses different file names for different workers? > > Alternatively, is there a way for all of the workers to use the same > log and spool directories without conflicts? Currently, if I do that, > I see workers crashing. > > Thanks, > > Raj From ehoward at bbg.gov Thu Feb 26 10:26:27 2015 From: ehoward at bbg.gov (Eric Howard) Date: Thu, 26 Feb 2015 18:26:27 +0000 Subject: [Bro] Log filtering a field re-ordering Message-ID: <1424975188028.12676@bbg.gov> Hi all, I have followed the instructions contained in https://www.bro.org/sphinx-git/frameworks/logging.html#filtering to create a new field output. I ahve noticed that the fields you choose to include cannot be be re-ordered for display. For example, if I put the 'ts' field in the first position like this: local filter: Log::Filter = [$name="orig-only", $path="origs", $include=set("id.orig_h","ts")]; the record displays with it in the first position. I assume this is because the include set is just a toggle that does not affect display order which is based on the field position in INFO. How to I re-order the the fields for display? Is this done ion the writer? Thanks! -- Eric -- -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150226/0414e7df/attachment.html From gfaulkner.nsm at gmail.com Thu Feb 26 14:13:56 2015 From: gfaulkner.nsm at gmail.com (Gary Faulkner) Date: Thu, 26 Feb 2015 16:13:56 -0600 Subject: [Bro] Bro 2.3.2-419 segfaults when using PF_RING 6.0.3 libpcap 1.6.2 and pfdnacluster_master on RHEL 6.6 In-Reply-To: <54ECE63E.201@gmail.com> References: <54ECC1BF.8070106@gmail.com> <54ECE63E.201@gmail.com> Message-ID: <54EF9AA4.2060401@gmail.com> All, A few other folks reported similar segfault issues to the PF_RING team both with standard PF_RING and DNA/ZC. After some troubleshooting and debugging they were able to to issue a patch (in SVN build 9021) that at least in initial testing seems to have resolved the segfault issue. Bro appears to now work segfault free using PF_RING (6.0.3 build 9021) both without DNA/ZC and with DNA using RSS. I'm still seeing a separate issue I'm following up with them on concerning not being able to map more than 10 app instances when using libzero's pfdnacluster_master script for load-balancing on host. Regards, Gary On 2/24/2015 2:59 PM, Gary Faulkner wrote: > A couple folks have suggested I run this with gdb and get a backtrace > to post here. Here is a quick gdb session with a backtrace of when I > run bro -i dnacluster:21 at 0: > > # gdb /nsm/bro/bin/bro > GNU gdb (GDB) SLES Expanded Support platform (7.2-75.el6) > Copyright (C) 2010 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show > copying" > and "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux". > For bug reporting instructions, please see: > ... > Reading symbols from /nsm/bro/bin/bro...done. > (gdb) run -i dnacluster:21 at 0 > Starting program: /nsm/bro/bin/bro -i dnacluster:21 at 0 > [Thread debugging using libthread_db enabled] > listening on dnacluster:21 at 0, capture length 8192 bytes > > [New Thread 0x7fff20fd0700 (LWP 36513)] > [New Thread 0x7fff1bfff700 (LWP 36514)] > [New Thread 0x7fff1b5fe700 (LWP 36515)] > [New Thread 0x7fff1abfd700 (LWP 36516)] > [New Thread 0x7fff1a1fc700 (LWP 36517)] > [New Thread 0x7fff197fb700 (LWP 36518)] > [New Thread 0x7fff18dfa700 (LWP 36519)] > [New Thread 0x7fff03fff700 (LWP 36520)] > [New Thread 0x7fff035fe700 (LWP 36521)] > [New Thread 0x7fff02bfd700 (LWP 36522)] > [New Thread 0x7fff021fc700 (LWP 36523)] > [New Thread 0x7fff017fb700 (LWP 36524)] > > Program received signal SIGSEGV, Segmentation fault. > 0x00007ffff7959506 in pcap_read_packet (handle=0x2631640, > callback=0x7ffff795d720 , userdata=0x7fffffffda20 > "p\025c\002") at ./pcap-linux.c:1807 > 1807 ./pcap-linux.c: No such file or directory. > in ./pcap-linux.c > Missing separate debuginfos, use: debuginfo-install > GeoIP-1.5.1-5.el6.x86_64 glibc-2.12-1.149.el6_6.5.x86_64 > keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-33.el6.x86_64 > libcom_err-1.41.12-21.el6.x86_64 libgcc-4.4.7-11.el6.x86_64 > libselinux-2.0.94-5.8.el6.x86_64 libstdc++-4.4.7-11.el6.x86_64 > numactl-2.0.9-2.el6.x86_64 openssl-1.0.1e-30.el6_6.5.x86_64 > zlib-1.2.3-29.el6.x86_64 > (gdb) bt > #0 0x00007ffff7959506 in pcap_read_packet (handle=0x2631640, > callback=0x7ffff795d720 , userdata=0x7fffffffda20 > "p\025c\002") at ./pcap-linux.c:1807 > #1 0x00007ffff795d79b in pcap_next (p=, h= optimized out>) at ./pcap.c:218 > #2 0x0000000000a4a490 in iosource::pcap::PcapSource::ExtractNextPacket > (this=0x2631430, pkt=0x2631468) at/nsm/bro/git/bro2.3-419/bro/ > src/iosource/pcap/Source.cc:151 > #3 0x0000000000a7580c in iosource::PktSrc::ExtractNextPacketInternal > (this=0x2631430) at > /nsm/bro/git/bro2.3-419/bro/src/iosource/PktSrc.cc:432 > #4 0x0000000000a7511b in iosource::PktSrc::NextTimestamp > (this=0x2631430, local_network_time=0x7fffffffdcb8) at > /nsm/bro/git/bro2.3-419/bro/src/iosource/PktSrc.cc:241 > #5 0x0000000000a71193 in iosource::Manager::FindSoonest (this=0xf29bc0, > ts=0x7fffffffddc8) at/nsm/bro/git/bro2.3-419/bro/ > src/iosource/Manager.cc:82 > #6 0x00000000007895d1 in net_run () at/nsm/bro/git/bro2.3-419/bro/ > src/Net.cc:301 > #7 0x00000000006d8ed7 in main (argc=3, argv=0x7fffffffe498) at > /nsm/bro/git/bro2.3-419/bro/src/main.cc:1200 > > On 2/24/2015 1:20 PM, John Donnelly wrote: >> Can you use gdb to get a backstrace ? >> >> ... >> >> ---------- Forwarded message ---------- >> From: Gary Faulkner >> Date: Tue, Feb 24, 2015 at 12:23 PM >> Subject: [Bro] Bro 2.3.2-419 segfaults when using PF_RING 6.0.3 libpcap >> 1.6.2 and pfdnacluster_master on RHEL 6.6 >> To: "bro at bro.org List" >> >> >> Hello, >> >> I?m having trouble getting Bro to run with PF_RING after updating >> from RHEL >> 6.5 to RHEL 6.6. The PF_RING aware drivers (DNA/ZC etc) in the ?stable? >> 6.0.2 branch of PF_RING don?t appear to compile correctly on RHEL 6.6, >> which necessitated a move to the latest 6.0.3 development branch >> (rev.9009). This version compiles fine and I have it working with both >> Suricata and nprobe, but can?t get it working with Bro. Bro doesn?t >> seem to >> be able to open the dnacluster:21 at 0 etc interfaces with the new version. >> Specifically bro segfaults when calling the PF_RING version of >> libpcap.so.1.6.2, which is a new version of libpcap in 6.0.3. Previously >> libpcap was 1.1.1. I have also tried to compile PF_RING 6.0.2 stable on >> RHEL 6.6 with the newer drivers, but the version of pfdnacluster_master >> that ships with PF_RING 6.0.2 stable (that uses the older libpcap) will >> silently crash on RHEL 6.6. I?ve attached the output of a broctl diag to >> this email. Typically when I've seen an error where bro can?t listen on >> dnacluster in the past it has been due to the interface already being in >> use, bro not being able to find pfring, or not compiling against the >> correct libpcap. I?ve verified this isn?t the case to the best of my >> ability (no other libpcap on the system, fresh dna driver load and >> instance >> of pfdnaclster_master, pfring in $PATH etc). I?ve also verified that >> I can >> see packets on the dnacluster interfaces by testing with pfcount. It >> looks >> like perhaps bro doesn?t like the new version of libpcap. I have tried >> compiling and running bro with debugging enabled, but bro seems to >> crash on >> the workers without generating anything in the various debug.log >> files. Any >> thoughts? >> >> Here are example error messages from /var/log/messages: >> >> kernel: bro[1653]: segfault at 1371670 ip 00007f5a9e7f0660 sp >> 00007fff8714b300 error 4 in libpcap.so.1.6.2[7f5a9e7d9000+90000] >> kernel: bro[1643]: segfault at 1371670 ip 00007ff16d19b660 sp >> 00007fff81eea9a0 error 4 in libpcap.so.1.6.2[7ff16d184000+90000] >> kernel: bro[1656]: segfault at 1371670 ip 00007fcf3c6cf660 sp >> 00007fff3e1789b0 error 4 in libpcap.so.1.6.2[7fcf3c6b8000+90000] >> kernel: bro[1644]: segfault at 1 ip 00007f5932268506 sp 00007fffcd3ea0b0 >> error 4 in libpcap.so.1.6.2[7f5932251000+90000] >> kernel: bro[1642]: segfault at 1 ip 00007ff3d1c83506 sp 00007fff468f4930 >> error 4 in libpcap.so.1.6.2[7ff3d1c6c000+90000] >> kernel: bro[1658]: segfault at 1371670 ip 00007f53584f2660 sp >> 00007ffff89515f0 error 4 in libpcap.so.1.6.2[7f53584db000+90000] >> kernel: bro[1652]: segfault at 1371670 ip 00007f158fbc7660 sp >> 00007fff14aa7e20 error 4 in libpcap.so.1.6.2[7f158fbb0000+90000] >> kernel: bro[1660]: segfault at 1371670 ip 00007f2fee8e7660 sp >> 00007ffff9dacaf0 error 4 in libpcap.so.1.6.2[7f2fee8d0000+90000] >> kernel: bro[1641]: segfault at 1 ip 00007f32fbc48506 sp 00007fff7d9b2a00 >> error 4 in libpcap.so.1.6.2[7f32fbc31000+90000] >> kernel: bro[1662]: segfault at b836210 ip 00007f5c9d669660 sp >> 00007fff71636fb0 error 4 in libpcap.so.1.6.2[7f5c9d652000+90000] >> kernel: bro[4220]: segfault at 1371670 ip 00007f6d35299660 sp >> 00007fff4d896940 error 4 in libpcap.so.1.6.2[7f6d35282000+90000] >> kernel: bro[4465]: segfault at 1371670 ip 00007f202ff75660 sp >> 00007fff04fff8c0 error 4 in libpcap.so.1.6.2[7f202ff5e000+90000] >> kernel: bro[4710]: segfault at 1371670 ip 00007fd8bc794660 sp >> 00007fff33041db0 error 4 in libpcap.so.1.6.2[7fd8bc77d000+90000] >> kernel: bro[7873]: segfault at 1371670 ip 00007ffc910f2660 sp >> 00007fff1b5ba1b0 error 4 in libpcap.so.1.6.2[7ffc910db000+90000] >> kernel: bro[8065]: segfault at 1371670 ip 00007ffaa5c8f660 sp >> 00007fff3cdde390 error 4 in libpcap.so.1.6.2[7ffaa5c78000+90000] >> kernel: bro[8257]: segfault at 63745e0 ip 00007ff913224660 sp >> 00007fff297ca2f0 error 4 in libpcap.so.1.6.2[7ff91320d000+90000] >> kernel: bro[8446]: segfault at 1371670 ip 00007f0a1c567660 sp >> 00007fffdf059910 error 4 in libpcap.so.1.6.2[7f0a1c550000+90000] >> kernel: bro[8638]: segfault at 1371670 ip 00007f50982af660 sp >> 00007fff703caa30 error 4 in libpcap.so.1.6.2[7f5098298000+90000] >> kernel: bro[8835]: segfault at 1371670 ip 00007f1b4acd2660 sp >> 00007fffacc16630 error 4 in libpcap.so.1.6.2[7f1b4acbb000+90000] >> kernel: bro[9036]: segfault at 1 ip 00007f10df91b506 sp 00007fff5ac3e320 >> error 4 in libpcap.so.1.6.2[7f10df904000+90000] >> >> Regards, >> Gary >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > From michalpurzynski1 at gmail.com Thu Feb 26 16:37:30 2015 From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=) Date: Fri, 27 Feb 2015 01:37:30 +0100 Subject: [Bro] Bro 2.3.2-419 segfaults when using PF_RING 6.0.3 libpcap 1.6.2 and pfdnacluster_master on RHEL 6.6 In-Reply-To: <54EF9AA4.2060401@gmail.com> References: <54ECC1BF.8070106@gmail.com> <54ECE63E.201@gmail.com> <54EF9AA4.2060401@gmail.com> Message-ID: Confirmed, pf_ring from SVN (with libpcap 1.6.x line), Bro 2.3.2 and no segfaults from over 10 hours now. Everything works in the AWS VM (so no DNA/ZC, but multiple workers). I had issues with release version of pf_ring (6.0.2) but 6.0.3 build 9021 is OK. On Thu, Feb 26, 2015 at 11:13 PM, Gary Faulkner wrote: > All, > > A few other folks reported similar segfault issues to the PF_RING team > both with standard PF_RING and DNA/ZC. After some troubleshooting and > debugging they were able to to issue a patch (in SVN build 9021) that at > least in initial testing seems to have resolved the segfault issue. Bro > appears to now work segfault free using PF_RING (6.0.3 build 9021) both > without DNA/ZC and with DNA using RSS. I'm still seeing a separate issue > I'm following up with them on concerning not being able to map more than > 10 app instances when using libzero's pfdnacluster_master script for > load-balancing on host. > > Regards, > Gary > > On 2/24/2015 2:59 PM, Gary Faulkner wrote: >> A couple folks have suggested I run this with gdb and get a backtrace >> to post here. Here is a quick gdb session with a backtrace of when I >> run bro -i dnacluster:21 at 0: >> >> # gdb /nsm/bro/bin/bro >> GNU gdb (GDB) SLES Expanded Support platform (7.2-75.el6) >> Copyright (C) 2010 Free Software Foundation, Inc. >> License GPLv3+: GNU GPL version 3 or later > html> >> This is free software: you are free to change and redistribute it. >> There is NO WARRANTY, to the extent permitted by law. Type "show >> copying" >> and "show warranty" for details. >> This GDB was configured as "x86_64-redhat-linux". >> For bug reporting instructions, please see: >> ... >> Reading symbols from /nsm/bro/bin/bro...done. >> (gdb) run -i dnacluster:21 at 0 >> Starting program: /nsm/bro/bin/bro -i dnacluster:21 at 0 >> [Thread debugging using libthread_db enabled] >> listening on dnacluster:21 at 0, capture length 8192 bytes >> >> [New Thread 0x7fff20fd0700 (LWP 36513)] >> [New Thread 0x7fff1bfff700 (LWP 36514)] >> [New Thread 0x7fff1b5fe700 (LWP 36515)] >> [New Thread 0x7fff1abfd700 (LWP 36516)] >> [New Thread 0x7fff1a1fc700 (LWP 36517)] >> [New Thread 0x7fff197fb700 (LWP 36518)] >> [New Thread 0x7fff18dfa700 (LWP 36519)] >> [New Thread 0x7fff03fff700 (LWP 36520)] >> [New Thread 0x7fff035fe700 (LWP 36521)] >> [New Thread 0x7fff02bfd700 (LWP 36522)] >> [New Thread 0x7fff021fc700 (LWP 36523)] >> [New Thread 0x7fff017fb700 (LWP 36524)] >> >> Program received signal SIGSEGV, Segmentation fault. >> 0x00007ffff7959506 in pcap_read_packet (handle=0x2631640, >> callback=0x7ffff795d720 , userdata=0x7fffffffda20 >> "p\025c\002") at ./pcap-linux.c:1807 >> 1807 ./pcap-linux.c: No such file or directory. >> in ./pcap-linux.c >> Missing separate debuginfos, use: debuginfo-install >> GeoIP-1.5.1-5.el6.x86_64 glibc-2.12-1.149.el6_6.5.x86_64 >> keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-33.el6.x86_64 >> libcom_err-1.41.12-21.el6.x86_64 libgcc-4.4.7-11.el6.x86_64 >> libselinux-2.0.94-5.8.el6.x86_64 libstdc++-4.4.7-11.el6.x86_64 >> numactl-2.0.9-2.el6.x86_64 openssl-1.0.1e-30.el6_6.5.x86_64 >> zlib-1.2.3-29.el6.x86_64 >> (gdb) bt >> #0 0x00007ffff7959506 in pcap_read_packet (handle=0x2631640, >> callback=0x7ffff795d720 , userdata=0x7fffffffda20 >> "p\025c\002") at ./pcap-linux.c:1807 >> #1 0x00007ffff795d79b in pcap_next (p=, h=> optimized out>) at ./pcap.c:218 >> #2 0x0000000000a4a490 in iosource::pcap::PcapSource::ExtractNextPacket >> (this=0x2631430, pkt=0x2631468) at/nsm/bro/git/bro2.3-419/bro/ >> src/iosource/pcap/Source.cc:151 >> #3 0x0000000000a7580c in iosource::PktSrc::ExtractNextPacketInternal >> (this=0x2631430) at >> /nsm/bro/git/bro2.3-419/bro/src/iosource/PktSrc.cc:432 >> #4 0x0000000000a7511b in iosource::PktSrc::NextTimestamp >> (this=0x2631430, local_network_time=0x7fffffffdcb8) at >> /nsm/bro/git/bro2.3-419/bro/src/iosource/PktSrc.cc:241 >> #5 0x0000000000a71193 in iosource::Manager::FindSoonest (this=0xf29bc0, >> ts=0x7fffffffddc8) at/nsm/bro/git/bro2.3-419/bro/ >> src/iosource/Manager.cc:82 >> #6 0x00000000007895d1 in net_run () at/nsm/bro/git/bro2.3-419/bro/ >> src/Net.cc:301 >> #7 0x00000000006d8ed7 in main (argc=3, argv=0x7fffffffe498) at >> /nsm/bro/git/bro2.3-419/bro/src/main.cc:1200 >> >> On 2/24/2015 1:20 PM, John Donnelly wrote: >>> Can you use gdb to get a backstrace ? >>> >>> ... >>> >>> ---------- Forwarded message ---------- >>> From: Gary Faulkner >>> Date: Tue, Feb 24, 2015 at 12:23 PM >>> Subject: [Bro] Bro 2.3.2-419 segfaults when using PF_RING 6.0.3 libpcap >>> 1.6.2 and pfdnacluster_master on RHEL 6.6 >>> To: "bro at bro.org List" >>> >>> >>> Hello, >>> >>> I?m having trouble getting Bro to run with PF_RING after updating >>> from RHEL >>> 6.5 to RHEL 6.6. The PF_RING aware drivers (DNA/ZC etc) in the ?stable? >>> 6.0.2 branch of PF_RING don?t appear to compile correctly on RHEL 6.6, >>> which necessitated a move to the latest 6.0.3 development branch >>> (rev.9009). This version compiles fine and I have it working with both >>> Suricata and nprobe, but can?t get it working with Bro. Bro doesn?t >>> seem to >>> be able to open the dnacluster:21 at 0 etc interfaces with the new version. >>> Specifically bro segfaults when calling the PF_RING version of >>> libpcap.so.1.6.2, which is a new version of libpcap in 6.0.3. Previously >>> libpcap was 1.1.1. I have also tried to compile PF_RING 6.0.2 stable on >>> RHEL 6.6 with the newer drivers, but the version of pfdnacluster_master >>> that ships with PF_RING 6.0.2 stable (that uses the older libpcap) will >>> silently crash on RHEL 6.6. I?ve attached the output of a broctl diag to >>> this email. Typically when I've seen an error where bro can?t listen on >>> dnacluster in the past it has been due to the interface already being in >>> use, bro not being able to find pfring, or not compiling against the >>> correct libpcap. I?ve verified this isn?t the case to the best of my >>> ability (no other libpcap on the system, fresh dna driver load and >>> instance >>> of pfdnaclster_master, pfring in $PATH etc). I?ve also verified that >>> I can >>> see packets on the dnacluster interfaces by testing with pfcount. It >>> looks >>> like perhaps bro doesn?t like the new version of libpcap. I have tried >>> compiling and running bro with debugging enabled, but bro seems to >>> crash on >>> the workers without generating anything in the various debug.log >>> files. Any >>> thoughts? >>> >>> Here are example error messages from /var/log/messages: >>> >>> kernel: bro[1653]: segfault at 1371670 ip 00007f5a9e7f0660 sp >>> 00007fff8714b300 error 4 in libpcap.so.1.6.2[7f5a9e7d9000+90000] >>> kernel: bro[1643]: segfault at 1371670 ip 00007ff16d19b660 sp >>> 00007fff81eea9a0 error 4 in libpcap.so.1.6.2[7ff16d184000+90000] >>> kernel: bro[1656]: segfault at 1371670 ip 00007fcf3c6cf660 sp >>> 00007fff3e1789b0 error 4 in libpcap.so.1.6.2[7fcf3c6b8000+90000] >>> kernel: bro[1644]: segfault at 1 ip 00007f5932268506 sp 00007fffcd3ea0b0 >>> error 4 in libpcap.so.1.6.2[7f5932251000+90000] >>> kernel: bro[1642]: segfault at 1 ip 00007ff3d1c83506 sp 00007fff468f4930 >>> error 4 in libpcap.so.1.6.2[7ff3d1c6c000+90000] >>> kernel: bro[1658]: segfault at 1371670 ip 00007f53584f2660 sp >>> 00007ffff89515f0 error 4 in libpcap.so.1.6.2[7f53584db000+90000] >>> kernel: bro[1652]: segfault at 1371670 ip 00007f158fbc7660 sp >>> 00007fff14aa7e20 error 4 in libpcap.so.1.6.2[7f158fbb0000+90000] >>> kernel: bro[1660]: segfault at 1371670 ip 00007f2fee8e7660 sp >>> 00007ffff9dacaf0 error 4 in libpcap.so.1.6.2[7f2fee8d0000+90000] >>> kernel: bro[1641]: segfault at 1 ip 00007f32fbc48506 sp 00007fff7d9b2a00 >>> error 4 in libpcap.so.1.6.2[7f32fbc31000+90000] >>> kernel: bro[1662]: segfault at b836210 ip 00007f5c9d669660 sp >>> 00007fff71636fb0 error 4 in libpcap.so.1.6.2[7f5c9d652000+90000] >>> kernel: bro[4220]: segfault at 1371670 ip 00007f6d35299660 sp >>> 00007fff4d896940 error 4 in libpcap.so.1.6.2[7f6d35282000+90000] >>> kernel: bro[4465]: segfault at 1371670 ip 00007f202ff75660 sp >>> 00007fff04fff8c0 error 4 in libpcap.so.1.6.2[7f202ff5e000+90000] >>> kernel: bro[4710]: segfault at 1371670 ip 00007fd8bc794660 sp >>> 00007fff33041db0 error 4 in libpcap.so.1.6.2[7fd8bc77d000+90000] >>> kernel: bro[7873]: segfault at 1371670 ip 00007ffc910f2660 sp >>> 00007fff1b5ba1b0 error 4 in libpcap.so.1.6.2[7ffc910db000+90000] >>> kernel: bro[8065]: segfault at 1371670 ip 00007ffaa5c8f660 sp >>> 00007fff3cdde390 error 4 in libpcap.so.1.6.2[7ffaa5c78000+90000] >>> kernel: bro[8257]: segfault at 63745e0 ip 00007ff913224660 sp >>> 00007fff297ca2f0 error 4 in libpcap.so.1.6.2[7ff91320d000+90000] >>> kernel: bro[8446]: segfault at 1371670 ip 00007f0a1c567660 sp >>> 00007fffdf059910 error 4 in libpcap.so.1.6.2[7f0a1c550000+90000] >>> kernel: bro[8638]: segfault at 1371670 ip 00007f50982af660 sp >>> 00007fff703caa30 error 4 in libpcap.so.1.6.2[7f5098298000+90000] >>> kernel: bro[8835]: segfault at 1371670 ip 00007f1b4acd2660 sp >>> 00007fffacc16630 error 4 in libpcap.so.1.6.2[7f1b4acbb000+90000] >>> kernel: bro[9036]: segfault at 1 ip 00007f10df91b506 sp 00007fff5ac3e320 >>> error 4 in libpcap.so.1.6.2[7f10df904000+90000] >>> >>> Regards, >>> Gary >>> >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >> >> > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From gfaulkner.nsm at gmail.com Fri Feb 27 08:43:04 2015 From: gfaulkner.nsm at gmail.com (Gary Faulkner) Date: Fri, 27 Feb 2015 10:43:04 -0600 Subject: [Bro] Bro 2.3.2-419 segfaults when using PF_RING 6.0.3 libpcap 1.6.2 and pfdnacluster_master on RHEL 6.6 In-Reply-To: References: <54ECC1BF.8070106@gmail.com> <54ECE63E.201@gmail.com> <54EF9AA4.2060401@gmail.com> Message-ID: <54F09E98.2010008@gmail.com> As a follow up to the second issue I was seeing. Alfredo (from NTOP) suggested that pfdnacluster_master was not allowing me to listen with more than 10 app instances possibly due to the name being too long when using the long format dnacluster:21 at 10 as opposed to dnacl:21 at 10. Bro used to work fine with the long name, so perhaps this is some change in PF_RING. Using the short name allowed Bro to bind workers to dnacl:21 at 10 and greater, but this seems to cause broctl to not call capstats properly (for a dnacluster). When using dnacluster:21 'broctl capstats' had some logic that would trigger broctl to call a single instance of capstats for each worker node as app2 on the dnacluster, where app2 had a single queue with a full copy of the traffic, so it wouldn't get a 'no such device message'. This doesn't seem to work when using the short name version of 'dnacl', and instead broctl calls capstats for each individual worker and since these are already bound to the app1 queues there is nothing for capstats to listen on. Regards, Gary On Thu, Feb 26, 2015 at 11:13 PM, Gary Faulkner wrote: >> I'm still seeing a separate issue >> I'm following up with them on concerning not being able to map more than >> 10 app instances when using libzero's pfdnacluster_master script for >> load-balancing on host. >> >> Regards, >> Gary >> >> On 2/24/2015 2:59 PM, Gary Faulkner wrote: >>> A couple folks have suggested I run this with gdb and get a backtrace >>> to post here. Here is a quick gdb session with a backtrace of when I >>> run bro -i dnacluster:21 at 0: >>> From doris at bro.org Fri Feb 27 10:22:40 2015 From: doris at bro.org (Doris Schioberg) Date: Fri, 27 Feb 2015 10:22:40 -0800 Subject: [Bro] Bro Monthly #4 is here Message-ID: <54F0B5F0.4060002@bro.org> Get the latest Bro news here: http://blog.bro.org/2015/02/bro-monthly-4.html - The Bro Team -- Doris Schioberg Bro Outreach, Training, and Education Coordinator International Computer Science Institute (ICSI Berkeley) Phone: +1 (510) 289-8406 * doris at bro.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 882 bytes Desc: OpenPGP digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150227/724d9144/attachment-0001.bin From dnthayer at illinois.edu Fri Feb 27 12:37:04 2015 From: dnthayer at illinois.edu (Daniel Thayer) Date: Fri, 27 Feb 2015 14:37:04 -0600 Subject: [Bro] Bro 2.3.2-419 segfaults when using PF_RING 6.0.3 libpcap 1.6.2 and pfdnacluster_master on RHEL 6.6 In-Reply-To: <54F09E98.2010008@gmail.com> References: <54ECC1BF.8070106@gmail.com> <54ECE63E.201@gmail.com> <54EF9AA4.2060401@gmail.com> <54F09E98.2010008@gmail.com> Message-ID: <54F0D570.7050204@illinois.edu> Thanks for reporting this issue. I've made a small change to broctl so that it now recognizes the shorter name "dnacl". -Daniel On 02/27/2015 10:43 AM, Gary Faulkner wrote: > As a follow up to the second issue I was seeing. Alfredo (from NTOP) > suggested that pfdnacluster_master was not allowing me to listen with > more than 10 app instances possibly due to the name being too long when > using the long format dnacluster:21 at 10 as opposed to dnacl:21 at 10. Bro > used to work fine with the long name, so perhaps this is some change in > PF_RING. Using the short name allowed Bro to bind workers to dnacl:21 at 10 > and greater, but this seems to cause broctl to not call capstats > properly (for a dnacluster). When using dnacluster:21 'broctl capstats' > had some logic that would trigger broctl to call a single instance of > capstats for each worker node as app2 on the dnacluster, where app2 had > a single queue with a full copy of the traffic, so it wouldn't get a 'no > such device message'. This doesn't seem to work when using the short > name version of 'dnacl', and instead broctl calls capstats for each > individual worker and since these are already bound to the app1 queues > there is nothing for capstats to listen on. > > Regards, > Gary > From gfaulkner.nsm at gmail.com Fri Feb 27 12:57:40 2015 From: gfaulkner.nsm at gmail.com (Gary Faulkner) Date: Fri, 27 Feb 2015 14:57:40 -0600 Subject: [Bro] Bro 2.3.2-419 segfaults when using PF_RING 6.0.3 libpcap 1.6.2 and pfdnacluster_master on RHEL 6.6 In-Reply-To: <54F0D570.7050204@illinois.edu> References: <54ECC1BF.8070106@gmail.com> <54ECE63E.201@gmail.com> <54EF9AA4.2060401@gmail.com> <54F09E98.2010008@gmail.com> <54F0D570.7050204@illinois.edu> Message-ID: <54F0DA44.1000503@gmail.com> Thanks, Daniel! I can confirm that the patch fixed the issue for 'broctl capstats' when using the short name dnacl for an interface for me. Regards, Gary On 2/27/2015 2:37 PM, Daniel Thayer wrote: > Thanks for reporting this issue. I've made a small change > to broctl so that it now recognizes the shorter name "dnacl". > > -Daniel > > > On 02/27/2015 10:43 AM, Gary Faulkner wrote: >> As a follow up to the second issue I was seeing. Alfredo (from NTOP) >> suggested that pfdnacluster_master was not allowing me to listen with >> more than 10 app instances possibly due to the name being too long when >> using the long format dnacluster:21 at 10 as opposed to dnacl:21 at 10. Bro >> used to work fine with the long name, so perhaps this is some change in >> PF_RING. Using the short name allowed Bro to bind workers to dnacl:21 at 10 >> and greater, but this seems to cause broctl to not call capstats >> properly (for a dnacluster). When using dnacluster:21 'broctl capstats' >> had some logic that would trigger broctl to call a single instance of >> capstats for each worker node as app2 on the dnacluster, where app2 had >> a single queue with a full copy of the traffic, so it wouldn't get a 'no >> such device message'. This doesn't seem to work when using the short >> name version of 'dnacl', and instead broctl calls capstats for each >> individual worker and since these are already bound to the app1 queues >> there is nothing for capstats to listen on. >> >> Regards, >> Gary >>