From zeutech at gmail.com Tue Aug 1 02:56:34 2017 From: zeutech at gmail.com (iraj norouzi) Date: Tue, 1 Aug 2017 14:26:34 +0430 Subject: [Bro] using netmap framework of freebsd for installing bro Message-ID: hi everybody i try to install bro on freebsd and because of 10G interface and traffic i need to use of netmap framework of freebsd but firstly after upgrading freebsd port, it didn't upgrade to bro 2.5.1 so i had to download source of 2.5.1 and use it for installation secondly for installing bro with netmap framework i found Michael Shirk pdf which mention configure bro with netmap by --witch-netmap=/usr/src command but there is no --witch-netmap option for bro configuration, so please help me to install bro on freebsd 11 with netmap framework. *Regards,Iraj Norouzi* *+989122494558* -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170801/8602b307/attachment.html From vikrambasu059 at gmail.com Tue Aug 1 05:44:31 2017 From: vikrambasu059 at gmail.com (Vikram Basu) Date: Tue, 1 Aug 2017 18:14:31 +0530 Subject: [Bro] IP address in addition to MAC and DHCP_host_name Message-ID: <598077b0.450b620a.d4e9.07e8@mx.google.com> Hi, I saw that the known_devices.log has timestamp, mac address and dhcp_host_name. I want to add the assigned IP address field to this log as well. I thought changing the known-devices-and-hostname.bro script in the following manner would add the field to the log. But it does not seem to work. What am I doing wrong ? Regards Vikram Basu Known-devices-and-hostnames.bro ------------------------------------------------- @load policy/misc/known-devices module Known; export { redef record DevicesInfo += { ## The value of the DHCP host name option, if seen. dhcp_host_name: string &log &optional; }; } event dhcp_request(c: connection, msg: dhcp_msg, req_addr: addr, serv_addr: addr, host_name: string) { if ( msg$h_addr == "" ) return; if ( msg$h_addr !in known_devices ) { add known_devices[msg$h_addr]; Log::write(Known::DEVICES_LOG, [$ts=network_time(), $mac=msg$h_addr, $dhcp_host_name=host_name, $assigned_ip=msg$yiaddr]); } } event dhcp_inform(c: connection, msg: dhcp_msg, host_name: string) { if ( msg$h_addr == "" ) return; if ( msg$h_addr !in known_devices ) { add known_devices[msg$h_addr]; Log::write(Known::DEVICES_LOG, [$ts=network_time(), $mac=msg$h_addr, $dhcp_host_name=host_name, $assigned_ip=msg$yiaddr]); } } -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170801/0283844e/attachment-0001.html From seth at corelight.com Tue Aug 1 05:47:16 2017 From: seth at corelight.com (Seth Hall) Date: Tue, 1 Aug 2017 08:47:16 -0400 Subject: [Bro] using netmap framework of freebsd for installing bro In-Reply-To: References: Message-ID: This is surprisingly easy now! Through Corelight we sponsored several development efforts in conjunction with the Netmap developers. One of the biggest things missing is a way to load balance the traffic, but the netmap respository has a tool in it now for doing that named "lb" (load balancer). If you clone the netmap repository and build that tool you'll be able to balance traffic from a single interface out to a number of Bro processes. Conveniently it also has very nice logs and can do buffering to help you weather traffic spikes. Here's the help output from lb.... usage: lb [options] where options are: -h view help text -i iface interface name (required) -p [prefix:]npipes add a new group of output pipes -B nbufs number of extra buffers (default: 0) -b batch batch size (default: 2048) -w seconds wait for link up (default: 2) -W enable busy waiting. this will run your CPU at 100% -s seconds seconds between syslog stats messages (default: 0) -o seconds seconds between stdout stats messages (default: 0) You would normally run it like this... lb -i -p -o 60 You give it the interface you are sniffing, how many Bro workers you are going to run and "-o 60" makes it write logs to stdout every 60 seconds. I need to create a bro-pkg with the netmap plugin that will make this all a bit easier too. .Seth On Tue, Aug 1, 2017 at 5:56 AM, iraj norouzi wrote: > hi everybody > i try to install bro on freebsd and because of 10G interface and traffic i > need to use of netmap framework of freebsd but firstly after upgrading > freebsd port, it didn't upgrade to bro 2.5.1 so i had to download source of > 2.5.1 and use it for installation secondly for installing bro with netmap > framework i found Michael Shirk pdf which mention configure bro with netmap > by --witch-netmap=/usr/src command but there is no --witch-netmap option for > bro configuration, so please help me to install bro on freebsd 11 with > netmap framework. > Regards, > Iraj Norouzi > +989122494558 > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From zeutech at gmail.com Tue Aug 1 07:13:57 2017 From: zeutech at gmail.com (iraj norouzi) Date: Tue, 1 Aug 2017 18:43:57 +0430 Subject: [Bro] using netmap framework of freebsd for installing bro In-Reply-To: References: Message-ID: thank for your reply Seth as mentioned at the https://www.bro.org/sphinx/components/bro-plugins/ netmap/README.html for using netmap framework with Bro if there is no customization on netmap you don't need to use --with-netmap option and if you customized netmap it needs to assign netmap option, but when i use ./configure --help i don't find any --with-netmap option and because of that i installed Bro and test it by bro -N Bro::Netmap command on the URL i gave "error in /usr/local/bro/share/bro/base/init-bare.bro, line 1: plugin Bro::Netmap is not available fatal error in /usr/local/bro/share/bro/base/init-bare.bro, line 1: Failed to activate requested dynamic plugin(s)" while when i use tcpdump -i netmap:ocs0 it recognized netmap. so please can you give me the manual for installing bro with netmap and test the functionality of that. as i read on the internet i can run workers equal to my CPU cores and i have 12 CPU core. *Regards,Iraj Norouzi* *+989122494558* On Tue, Aug 1, 2017 at 5:17 PM, Seth Hall wrote: > This is surprisingly easy now! Through Corelight we sponsored several > development efforts in conjunction with the Netmap developers. One of > the biggest things missing is a way to load balance the traffic, but > the netmap respository has a tool in it now for doing that named "lb" > (load balancer). If you clone the netmap repository and build that > tool you'll be able to balance traffic from a single interface out to > a number of Bro processes. Conveniently it also has very nice logs > and can do buffering to help you weather traffic spikes. > > Here's the help output from lb.... > > usage: lb [options] > where options are: > -h view help text > -i iface interface name (required) > -p [prefix:]npipes add a new group of output pipes > -B nbufs number of extra buffers (default: 0) > -b batch batch size (default: 2048) > -w seconds wait for link up (default: 2) > -W enable busy waiting. this will run your CPU at 100% > -s seconds seconds between syslog stats messages (default: 0) > -o seconds seconds between stdout stats messages (default: 0) > > You would normally run it like this... > > lb -i -p -o 60 > > You give it the interface you are sniffing, how many Bro workers you > are going to run and "-o 60" makes it write logs to stdout every 60 > seconds. I need to create a bro-pkg with the netmap plugin that will > make this all a bit easier too. > > .Seth > > On Tue, Aug 1, 2017 at 5:56 AM, iraj norouzi wrote: > > hi everybody > > i try to install bro on freebsd and because of 10G interface and traffic > i > > need to use of netmap framework of freebsd but firstly after upgrading > > freebsd port, it didn't upgrade to bro 2.5.1 so i had to download source > of > > 2.5.1 and use it for installation secondly for installing bro with netmap > > framework i found Michael Shirk pdf which mention configure bro with > netmap > > by --witch-netmap=/usr/src command but there is no --witch-netmap option > for > > bro configuration, so please help me to install bro on freebsd 11 with > > netmap framework. > > Regards, > > Iraj Norouzi > > +989122494558 > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > -- > Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170801/45d4a9cc/attachment.html From shirkdog.bsd at gmail.com Tue Aug 1 07:57:25 2017 From: shirkdog.bsd at gmail.com (Michael Shirk) Date: Tue, 1 Aug 2017 10:57:25 -0400 Subject: [Bro] using netmap framework of freebsd for installing bro In-Reply-To: References: Message-ID: You will want to look at the how-to for the netmap plugin on GitHub as things have changed. However...the how-to has moved. Seth, where is that info for the plugins now that they are packages? -- Michael Shirk Daemon Security, Inc. https://www.daemon-security.com On Aug 1, 2017 10:16 AM, "iraj norouzi" wrote: > thank for your reply Seth > as mentioned at the https://www.bro.org/sphinx > /components/bro-plugins/netmap/README.html for using netmap framework > with Bro if there is no customization on netmap you don't need to use > --with-netmap option and if you customized netmap it needs to assign netmap > option, but when i use ./configure --help i don't find any --with-netmap > option and because of that i installed Bro and test it by bro -N > Bro::Netmap command on the URL i gave "error in > /usr/local/bro/share/bro/base/init-bare.bro, line 1: plugin Bro::Netmap > is not available > fatal error in /usr/local/bro/share/bro/base/init-bare.bro, line 1: > Failed to activate requested dynamic plugin(s)" while when i use tcpdump -i > netmap:ocs0 it recognized netmap. > so please can you give me the manual for installing bro with netmap and > test the functionality of that. > as i read on the internet i can run workers equal to my CPU cores and i > have 12 CPU core. > > > *Regards,Iraj Norouzi* > *+989122494558 <+98%20912%20249%204558>* > > On Tue, Aug 1, 2017 at 5:17 PM, Seth Hall wrote: > >> This is surprisingly easy now! Through Corelight we sponsored several >> development efforts in conjunction with the Netmap developers. One of >> the biggest things missing is a way to load balance the traffic, but >> the netmap respository has a tool in it now for doing that named "lb" >> (load balancer). If you clone the netmap repository and build that >> tool you'll be able to balance traffic from a single interface out to >> a number of Bro processes. Conveniently it also has very nice logs >> and can do buffering to help you weather traffic spikes. >> >> Here's the help output from lb.... >> >> usage: lb [options] >> where options are: >> -h view help text >> -i iface interface name (required) >> -p [prefix:]npipes add a new group of output pipes >> -B nbufs number of extra buffers (default: 0) >> -b batch batch size (default: 2048) >> -w seconds wait for link up (default: 2) >> -W enable busy waiting. this will run your CPU at >> 100% >> -s seconds seconds between syslog stats messages (default: 0) >> -o seconds seconds between stdout stats messages (default: 0) >> >> You would normally run it like this... >> >> lb -i -p -o 60 >> >> You give it the interface you are sniffing, how many Bro workers you >> are going to run and "-o 60" makes it write logs to stdout every 60 >> seconds. I need to create a bro-pkg with the netmap plugin that will >> make this all a bit easier too. >> >> .Seth >> >> On Tue, Aug 1, 2017 at 5:56 AM, iraj norouzi wrote: >> > hi everybody >> > i try to install bro on freebsd and because of 10G interface and >> traffic i >> > need to use of netmap framework of freebsd but firstly after upgrading >> > freebsd port, it didn't upgrade to bro 2.5.1 so i had to download >> source of >> > 2.5.1 and use it for installation secondly for installing bro with >> netmap >> > framework i found Michael Shirk pdf which mention configure bro with >> netmap >> > by --witch-netmap=/usr/src command but there is no --witch-netmap >> option for >> > bro configuration, so please help me to install bro on freebsd 11 with >> > netmap framework. >> > Regards, >> > Iraj Norouzi >> > +989122494558 <+98%20912%20249%204558> >> > >> > _______________________________________________ >> > Bro mailing list >> > bro at bro-ids.org >> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> >> >> -- >> Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com >> > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170801/36e7c63b/attachment-0001.html From jdopheid at illinois.edu Tue Aug 1 10:07:17 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Tue, 1 Aug 2017 17:07:17 +0000 Subject: [Bro] BroCon lightning talk sign-up Message-ID: <0CBD20AC-9BF2-40DF-B904-8170610319FF@illinois.edu> Hello Bro Community, In order to queue up requests to speak at the BroCon Lightning Talk session, we?ve created a Google form: https://goo.gl/forms/VetHBj0lQ2P2DZik2 We ask for your name, email address, organization, talk title, contact info, and a link to 1 or 2 slides to help us generate the slide deck. Lightning talk abstract: Have you contributed a package to the new Bro Package Manager? Or do you have something interesting to share related to Bro that doesn?t fit into a traditional presentation? Great! We?d love to hear from you. We have scheduled a 45-minute session for lightning talks. Be prepared to quickly identify your point, demonstrate it, and provide a link or contact info for later follow-up. We?d like to accommodate as many talks as possible so please limit your talk to less than 5 minutes. No commercially incentivized presentations, please. Register for BroCon: https://brocon2017.eventbrite.com/ Don?t forget to book your hotel: https://www.bro.org/community/brocon2017.html#hotelinformation The block of rooms expires on August 14th. See you in September, The Bro Project ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From zeutech at gmail.com Tue Aug 1 22:49:06 2017 From: zeutech at gmail.com (iraj norouzi) Date: Wed, 2 Aug 2017 10:19:06 +0430 Subject: [Bro] using netmap framework of freebsd for installing bro In-Reply-To: References: Message-ID: ?thank you Michael as you mentioned on the manual of installing bro on freebsd and on the bro guide on the bro website i tried to install bro with netmap by --with-netmap but there is no option for that so i installed bro and after that i installed netmap plugin for bro from /aux/plugins/netmap? folder and after that everything is OK, and i also used Seth manual for configuring node.cfg for workers, thank you so much Virus-free. www.avast.com <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> *Regards,Iraj Norouzi* *+989122494558* On Tue, Aug 1, 2017 at 7:27 PM, Michael Shirk wrote: > You will want to look at the how-to for the netmap plugin on GitHub as > things have changed. However...the how-to has moved. > > Seth, where is that info for the plugins now that they are packages? > > > -- > Michael Shirk > Daemon Security, Inc. > https://www.daemon-security.com > > On Aug 1, 2017 10:16 AM, "iraj norouzi" wrote: > >> thank for your reply Seth >> as mentioned at the https://www.bro.org/sphinx >> /components/bro-plugins/netmap/README.html for using netmap framework >> with Bro if there is no customization on netmap you don't need to use >> --with-netmap option and if you customized netmap it needs to assign netmap >> option, but when i use ./configure --help i don't find any --with-netmap >> option and because of that i installed Bro and test it by bro -N >> Bro::Netmap command on the URL i gave "error in >> /usr/local/bro/share/bro/base/init-bare.bro, line 1: plugin Bro::Netmap >> is not available >> fatal error in /usr/local/bro/share/bro/base/init-bare.bro, line 1: >> Failed to activate requested dynamic plugin(s)" while when i use tcpdump -i >> netmap:ocs0 it recognized netmap. >> so please can you give me the manual for installing bro with netmap and >> test the functionality of that. >> as i read on the internet i can run workers equal to my CPU cores and i >> have 12 CPU core. >> >> >> *Regards,Iraj Norouzi* >> *+989122494558 <+98%20912%20249%204558>* >> >> On Tue, Aug 1, 2017 at 5:17 PM, Seth Hall wrote: >> >>> This is surprisingly easy now! Through Corelight we sponsored several >>> development efforts in conjunction with the Netmap developers. One of >>> the biggest things missing is a way to load balance the traffic, but >>> the netmap respository has a tool in it now for doing that named "lb" >>> (load balancer). If you clone the netmap repository and build that >>> tool you'll be able to balance traffic from a single interface out to >>> a number of Bro processes. Conveniently it also has very nice logs >>> and can do buffering to help you weather traffic spikes. >>> >>> Here's the help output from lb.... >>> >>> usage: lb [options] >>> where options are: >>> -h view help text >>> -i iface interface name (required) >>> -p [prefix:]npipes add a new group of output pipes >>> -B nbufs number of extra buffers (default: 0) >>> -b batch batch size (default: 2048) >>> -w seconds wait for link up (default: 2) >>> -W enable busy waiting. this will run your CPU at >>> 100% >>> -s seconds seconds between syslog stats messages (default: 0) >>> -o seconds seconds between stdout stats messages (default: 0) >>> >>> You would normally run it like this... >>> >>> lb -i -p -o 60 >>> >>> You give it the interface you are sniffing, how many Bro workers you >>> are going to run and "-o 60" makes it write logs to stdout every 60 >>> seconds. I need to create a bro-pkg with the netmap plugin that will >>> make this all a bit easier too. >>> >>> .Seth >>> >>> On Tue, Aug 1, 2017 at 5:56 AM, iraj norouzi wrote: >>> > hi everybody >>> > i try to install bro on freebsd and because of 10G interface and >>> traffic i >>> > need to use of netmap framework of freebsd but firstly after upgrading >>> > freebsd port, it didn't upgrade to bro 2.5.1 so i had to download >>> source of >>> > 2.5.1 and use it for installation secondly for installing bro with >>> netmap >>> > framework i found Michael Shirk pdf which mention configure bro with >>> netmap >>> > by --witch-netmap=/usr/src command but there is no --witch-netmap >>> option for >>> > bro configuration, so please help me to install bro on freebsd 11 with >>> > netmap framework. >>> > Regards, >>> > Iraj Norouzi >>> > +989122494558 <+98%20912%20249%204558> >>> > >>> > _______________________________________________ >>> > Bro mailing list >>> > bro at bro-ids.org >>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >>> >>> >>> -- >>> Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com >>> >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170802/39483b84/attachment.html From jlcmoore at cs.washington.edu Wed Aug 2 10:46:31 2017 From: jlcmoore at cs.washington.edu (Jared Moore) Date: Wed, 2 Aug 2017 12:46:31 -0500 Subject: [Bro] Filter Questions Message-ID: <0E53DABE-64CE-424E-AC1F-9C997FFB9E3B@cs.washington.edu> Hi! I?m a masters student at the University of Washington and I?m setting up an installation to inform users of a space about digital privacy and teach them about threat modeling by displaying web sites requested in an open wifi network on a few displays. I have an openwrt router using port-mirroring to send a copy of all packets to my linux machine which is running bro to filter the headers and harvest just the source ip, host, uri, and user-agent, but I?m having trouble developing the proper bro code to filter out (ideally) all get requests besides the initial ones when a users clicks a link or types one in the address bar. The solution doesn?t need to be perfect, but I still need to narrow the scope dramatically. The following code is better than nothing, but it doesn?t filter out enough. I have a python script extracting the urls from the sql database and loading a few firefox browsers with a new url every couple of seconds and I want the urls queried to be visually similar to what the page a user requests to highlight the vulnerability of unencrypted traffic. I initially tried to extract the files from http connections and then load the html pages in the browsers, but I can?t seem to resolve the original names of the files appropriately. One suggestion I found was to use Xplico , but I couldn?t get that to work. I?m new to bro and appreciate any advice you have! Thanks, Jared @load base/protocols/http module HttpToSql; export { redef enum Log::ID += { LOG }; type Request: record { ts: string &log; source: addr &log; dest: addr &log; dest_port: port &log; method: string &log &optional; host: string &log &optional; uri: string &log &optional; url: string &log; referrer: string &log &optional; user_agent: string &log &optional; content_length: count &log &optional; basic_auth_user: string &log &optional; trans_depth: count &log; }; } event bro_init() { Log::create_stream(LOG, [$columns = Request]); local sql_filter: Log::Filter = [$name = "http-extracted-sqlite", $path = "/var/db/httptosql", $writer = Log::WRITER_SQLITE, $config = table(["tablename"] = "http")]; Log::add_filter(LOG, sql_filter); } event http_all_headers(c: connection, is_orig: bool, hlist: mime_header_list) { if (!is_orig) return; if ( !Site::is_local_addr(c$id$orig_h)) return; if ( !(/^[wW][wW][wW]/ in c$http$host)) return; if ( c$http$trans_depth > 1) return; local req: Request; req$ts = strftime("%Y/%m/%d %H:%M:%S", c$http$ts); req$trans_depth = c$http$trans_depth; req$source = c$id$orig_h; req$dest = c$id$resp_h; req$dest_port = c$id$resp_p; if (c$http?$method) req$method = c$http$method; if (c$http?$host) req$host = c$http$host; if (c$http?$uri) req$uri = c$http$uri; if (c$http?$referrer) req$referrer = c$http$referrer; if (c$http?$user_agent) req$user_agent = c$http$user_agent; if (c$http?$request_body_len) req$content_length = c$http$request_body_len; if (c$http?$username) req$basic_auth_user = c$http$username; req$url = HTTP::build_url_http(f$http); Log::write(LOG, req); } -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170802/026e5502/attachment-0001.html From apumphrey at bricata.com Wed Aug 2 16:02:23 2017 From: apumphrey at bricata.com (Adam Pumphrey) Date: Wed, 2 Aug 2017 23:02:23 +0000 Subject: [Bro] IP address in addition to MAC and DHCP_host_name In-Reply-To: <598077b0.450b620a.d4e9.07e8@mx.google.com> References: <598077b0.450b620a.d4e9.07e8@mx.google.com> Message-ID: <7C313EFF-2F22-4B21-8AF8-64910B29E558@bricata.com> You also need to add the assigned_ip field to the DevicesInfo record redef statement. From: on behalf of Vikram Basu Date: Tuesday, August 1, 2017 at 8:44 AM To: "bro at bro.org" Subject: [Bro] IP address in addition to MAC and DHCP_host_name Hi, I saw that the known_devices.log has timestamp, mac address and dhcp_host_name. I want to add the assigned IP address field to this log as well. I thought changing the known-devices-and-hostname.bro script in the following manner would add the field to the log. But it does not seem to work. What am I doing wrong ? Regards Vikram Basu Known-devices-and-hostnames.bro ------------------------------------------------- @load policy/misc/known-devices module Known; export { redef record DevicesInfo += { ## The value of the DHCP host name option, if seen. dhcp_host_name: string &log &optional; }; } event dhcp_request(c: connection, msg: dhcp_msg, req_addr: addr, serv_addr: addr, host_name: string) { if ( msg$h_addr == "" ) return; if ( msg$h_addr !in known_devices ) { add known_devices[msg$h_addr]; Log::write(Known::DEVICES_LOG, [$ts=network_time(), $mac=msg$h_addr, $dhcp_host_name=host_name, $assigned_ip=msg$yiaddr]); } } event dhcp_inform(c: connection, msg: dhcp_msg, host_name: string) { if ( msg$h_addr == "" ) return; if ( msg$h_addr !in known_devices ) { add known_devices[msg$h_addr]; Log::write(Known::DEVICES_LOG, [$ts=network_time(), $mac=msg$h_addr, $dhcp_host_name=host_name, $assigned_ip=msg$yiaddr]); } } -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170802/bb03b6f0/attachment.html From reswob10 at gmail.com Wed Aug 2 18:19:47 2017 From: reswob10 at gmail.com (craig bowser) Date: Wed, 2 Aug 2017 21:19:47 -0400 Subject: [Bro] Help with bro scripting exercise question Message-ID: Hello all, trying to learn bro scripting. I am working through the exercises from the 2011 workshop and I'm getting an error. I'm on this page: https://www.bro.org/bro-workshop-2011/exercises/notices/index.html I'm on Pt3 More Advanced Policy Notice running this script: const watched_servers: set[addr] = { 172.16.238.136, 172.16.238.168, } &redef; redef Notice::policy += { [$action = Notice::ACTION_ALARM, $pred(n: Notice::Info) = { return n$note == SSH::Login && n$id$resp_h in watched_servers; } ] }; And I'm getting an error that says #bro -r ssh.pcap local advancebro.bro error in ./advancebro.bro, line 10: unknown identifier SSH::Login, at or near "SSH::Login" Is the SSH::Login no longer a valid function? Thanks. Craig L Bowser ____________________________ This email is measured by size. Bits and bytes may have settled during transport. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170802/80dc555d/attachment.html From seth at corelight.com Wed Aug 2 18:30:18 2017 From: seth at corelight.com (Seth Hall) Date: Wed, 2 Aug 2017 21:30:18 -0400 Subject: [Bro] using netmap framework of freebsd for installing bro In-Reply-To: References: Message-ID: Sorry about that. Unfortunately I suggested it as things are in flux. :( I'm hoping in the next few days I can find some time to repackage the netmap plugin into a Bro package that people can install through the package manager. .Seth On Tue, Aug 1, 2017 at 10:57 AM, Michael Shirk wrote: > You will want to look at the how-to for the netmap plugin on GitHub as > things have changed. However...the how-to has moved. > > Seth, where is that info for the plugins now that they are packages? > > > -- > Michael Shirk > Daemon Security, Inc. > https://www.daemon-security.com > > On Aug 1, 2017 10:16 AM, "iraj norouzi" wrote: >> >> thank for your reply Seth >> as mentioned at the >> https://www.bro.org/sphinx/components/bro-plugins/netmap/README.html for >> using netmap framework with Bro if there is no customization on netmap you >> don't need to use --with-netmap option and if you customized netmap it needs >> to assign netmap option, but when i use ./configure --help i don't find any >> --with-netmap option and because of that i installed Bro and test it by bro >> -N Bro::Netmap command on the URL i gave "error in >> /usr/local/bro/share/bro/base/init-bare.bro, line 1: plugin Bro::Netmap is >> not available >> fatal error in /usr/local/bro/share/bro/base/init-bare.bro, line 1: Failed >> to activate requested dynamic plugin(s)" while when i use tcpdump -i >> netmap:ocs0 it recognized netmap. >> so please can you give me the manual for installing bro with netmap and >> test the functionality of that. >> as i read on the internet i can run workers equal to my CPU cores and i >> have 12 CPU core. >> >> Regards, >> Iraj Norouzi >> +989122494558 >> >> On Tue, Aug 1, 2017 at 5:17 PM, Seth Hall wrote: >>> >>> This is surprisingly easy now! Through Corelight we sponsored several >>> development efforts in conjunction with the Netmap developers. One of >>> the biggest things missing is a way to load balance the traffic, but >>> the netmap respository has a tool in it now for doing that named "lb" >>> (load balancer). If you clone the netmap repository and build that >>> tool you'll be able to balance traffic from a single interface out to >>> a number of Bro processes. Conveniently it also has very nice logs >>> and can do buffering to help you weather traffic spikes. >>> >>> Here's the help output from lb.... >>> >>> usage: lb [options] >>> where options are: >>> -h view help text >>> -i iface interface name (required) >>> -p [prefix:]npipes add a new group of output pipes >>> -B nbufs number of extra buffers (default: 0) >>> -b batch batch size (default: 2048) >>> -w seconds wait for link up (default: 2) >>> -W enable busy waiting. this will run your CPU at >>> 100% >>> -s seconds seconds between syslog stats messages (default: 0) >>> -o seconds seconds between stdout stats messages (default: 0) >>> >>> You would normally run it like this... >>> >>> lb -i -p -o 60 >>> >>> You give it the interface you are sniffing, how many Bro workers you >>> are going to run and "-o 60" makes it write logs to stdout every 60 >>> seconds. I need to create a bro-pkg with the netmap plugin that will >>> make this all a bit easier too. >>> >>> .Seth >>> >>> On Tue, Aug 1, 2017 at 5:56 AM, iraj norouzi wrote: >>> > hi everybody >>> > i try to install bro on freebsd and because of 10G interface and >>> > traffic i >>> > need to use of netmap framework of freebsd but firstly after upgrading >>> > freebsd port, it didn't upgrade to bro 2.5.1 so i had to download >>> > source of >>> > 2.5.1 and use it for installation secondly for installing bro with >>> > netmap >>> > framework i found Michael Shirk pdf which mention configure bro with >>> > netmap >>> > by --witch-netmap=/usr/src command but there is no --witch-netmap >>> > option for >>> > bro configuration, so please help me to install bro on freebsd 11 with >>> > netmap framework. >>> > Regards, >>> > Iraj Norouzi >>> > +989122494558 >>> > >>> > _______________________________________________ >>> > Bro mailing list >>> > bro at bro-ids.org >>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >>> >>> >>> -- >>> Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com >> >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From seth at corelight.com Wed Aug 2 18:54:50 2017 From: seth at corelight.com (Seth Hall) Date: Wed, 2 Aug 2017 21:54:50 -0400 Subject: [Bro] Help with bro scripting exercise question In-Reply-To: References: Message-ID: Sorry that code is out of date! Quite a few years have passed since that workshop. Are you just trying to go through the workshop code or are you actually trying to achieve a particular behavior? If you're trying to achieve a behavior, I can help you out with the more modern way to do that if you'd like. If you are just trying to go through the workshop code then I'd say to just skip that one. :) Have you look at the exercises on http://try.bro.org/? .Seth On Wed, Aug 2, 2017 at 9:19 PM, craig bowser wrote: > > Hello all, trying to learn bro scripting. I am working through the > exercises from the 2011 workshop and I'm getting an error. > > I'm on this page: > https://www.bro.org/bro-workshop-2011/exercises/notices/index.html > > I'm on Pt3 More Advanced Policy Notice running this script: > > const watched_servers: set[addr] = { > 172.16.238.136, > 172.16.238.168, > } &redef; > > redef Notice::policy += { > [$action = Notice::ACTION_ALARM, > $pred(n: Notice::Info) = > { > return n$note == SSH::Login && n$id$resp_h in watched_servers; > } > ] > }; > > > And I'm getting an error that says > > #bro -r ssh.pcap local advancebro.bro > error in ./advancebro.bro, line 10: unknown identifier SSH::Login, at or > near "SSH::Login" > > > Is the SSH::Login no longer a valid function? > > Thanks. > > > Craig L Bowser > ____________________________ > > This email is measured by size. Bits and bytes may have settled during > transport. > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From reswob10 at gmail.com Wed Aug 2 19:54:24 2017 From: reswob10 at gmail.com (craig bowser) Date: Wed, 2 Aug 2017 22:54:24 -0400 Subject: [Bro] Help with bro scripting exercise question In-Reply-To: References: Message-ID: I'm trying to do several things. My main goal is to learn how to script so I can build notices and alerts based on threats specific to my environment. Right now, I'm trying to figure out how to get bro to send an email when it generates a notice. I've been looking at this page: https://www.bro.org/sphinx-git/frameworks/notice.html and this page: https://www.bro.org/sphinx-git/scripts/base/frameworks/notice/main.bro.html and this email from the archive: http://mailman.icsi.berkeley.edu/pipermail/bro/2014-October/007621.html and this reference from a google search: https://books.google.com/books?id=TTIDAQAAQBAJ&pg=PA283&lpg=PA283&dq=Notice::mail_dest&source=bl&ots=Uw_GRZCI2Q&sig=lzDZWVnDNdfIuFaRP16OWgXvk-4&hl=en&sa=X&ved=0ahUKEwinzbCT97nVAhUC7IMKHaBzDjkQ6AEIPTAE But my attempt at putting all of that together using the example script from the aforementioned 2011 exercise isn't working... #redef Notice::alarmed_types += { #SSH::Password_Guessing #}; redef Notice::ignored_types += { }; redef Notice::mail_dest = "reswob10 at gmail.com"; hook Notice::policy(n: Notice::Info) { if ( n$note == SSH::Password_Guessing ) add n$actions[Notice::ACTION_EMAIL]; } Any direction you can provide would be helpful. And yes, I've gone through most of those exercises at try.bro.org, but they don't cover how to send notices via email. Thanks. Craig On 2 Aug 2017 9:55 pm, "Seth Hall" wrote: > Sorry that code is out of date! Quite a few years have passed since > that workshop. Are you just trying to go through the workshop code or > are you actually trying to achieve a particular behavior? If you're > trying to achieve a behavior, I can help you out with the more modern > way to do that if you'd like. If you are just trying to go through > the workshop code then I'd say to just skip that one. :) > > Have you look at the exercises on http://try.bro.org/? > > .Seth > > On Wed, Aug 2, 2017 at 9:19 PM, craig bowser wrote: > > > > Hello all, trying to learn bro scripting. I am working through the > > exercises from the 2011 workshop and I'm getting an error. > > > > I'm on this page: > > https://www.bro.org/bro-workshop-2011/exercises/notices/index.html > > > > I'm on Pt3 More Advanced Policy Notice running this script: > > > > const watched_servers: set[addr] = { > > 172.16.238.136, > > 172.16.238.168, > > } &redef; > > > > redef Notice::policy += { > > [$action = Notice::ACTION_ALARM, > > $pred(n: Notice::Info) = > > { > > return n$note == SSH::Login && n$id$resp_h in > watched_servers; > > } > > ] > > }; > > > > > > And I'm getting an error that says > > > > #bro -r ssh.pcap local advancebro.bro > > error in ./advancebro.bro, line 10: unknown identifier SSH::Login, at or > > near "SSH::Login" > > > > > > Is the SSH::Login no longer a valid function? > > > > Thanks. > > > > > > Craig L Bowser > > ____________________________ > > > > This email is measured by size. Bits and bytes may have settled during > > transport. > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > -- > Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170802/7e3a269f/attachment.html From adh.2234 at gmail.com Fri Aug 4 05:06:13 2017 From: adh.2234 at gmail.com (=?UTF-8?B?0JjQstCw0L0g0KDQsNGC0LrQuNC9?=) Date: Fri, 4 Aug 2017 15:06:13 +0300 Subject: [Bro] Escape sequences in BRO (\n) Message-ID: Hi. I got a trouble with escape sequences in Bro. When i write: print "Next words must be in a new line \n New line" or print "\n" \n will decoded in x0a or \n, but not decoded as a new. So there is a question: how can i make a new line in strings? F.e. i read several strings of data and need to from string like "hi, its a new line and it is too!" but only thing i have is "hi,\nits a new line\nand it is too!" Thanks for answer -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170804/3596eee2/attachment.html From anthony.kasza at gmail.com Fri Aug 4 06:48:39 2017 From: anthony.kasza at gmail.com (anthony kasza) Date: Fri, 4 Aug 2017 07:48:39 -0600 Subject: [Bro] Escape sequences in BRO (\n) In-Reply-To: References: Message-ID: Have you tried using the fmt function in conjunction with the print function? https://www.bro.org/sphinx/scripts/base/bif/bro.bif.bro.html#id-fmt -AK On Aug 4, 2017 6:09 AM, "???? ??????" wrote: > Hi. I got a trouble with escape sequences in Bro. > When i write: > print "Next words must be in a new line \n New line" > or > print "\n" > > \n will decoded in x0a or \n, but not decoded as a new. > > So there is a question: how can i make a new line in strings? F.e. i read > several strings of data and need to from string like > "hi, > its a new line > and it is too!" > > but only thing i have is "hi,\nits a new line\nand it is too!" > > > Thanks for answer > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170804/c8c64439/attachment.html From adh.2234 at gmail.com Fri Aug 4 07:04:07 2017 From: adh.2234 at gmail.com (=?UTF-8?B?0JjQstCw0L0g0KDQsNGC0LrQuNC9?=) Date: Fri, 4 Aug 2017 17:04:07 +0300 Subject: [Bro] Escape sequences in BRO (\n) In-Reply-To: References: Message-ID: Yeah. The code: local line = fmt("%s", "Hi \n all!"); print line; will print the Hi \n all! :( 2017-08-04 16:48 GMT+03:00 anthony kasza : > Have you tried using the fmt function in conjunction with the print > function? > https://www.bro.org/sphinx/scripts/base/bif/bro.bif.bro.html#id-fmt > > -AK > > On Aug 4, 2017 6:09 AM, "???? ??????" wrote: > >> Hi. I got a trouble with escape sequences in Bro. >> When i write: >> print "Next words must be in a new line \n New line" >> or >> print "\n" >> >> \n will decoded in x0a or \n, but not decoded as a new. >> >> So there is a question: how can i make a new line in strings? F.e. i read >> several strings of data and need to from string like >> "hi, >> its a new line >> and it is too!" >> >> but only thing i have is "hi,\nits a new line\nand it is too!" >> >> >> Thanks for answer >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170804/5565ccb9/attachment.html From adh.2234 at gmail.com Fri Aug 4 07:14:00 2017 From: adh.2234 at gmail.com (=?UTF-8?B?0JjQstCw0L0g0KDQsNGC0LrQuNC9?=) Date: Fri, 4 Aug 2017 17:14:00 +0300 Subject: [Bro] Escape sequences in BRO (\n) In-Reply-To: References: Message-ID: sorry, it will print Hi \x0a all! And if slash is double: fmt("%s", "Hi \\n all!"); it will print Hi \n all! 2017-08-04 17:04 GMT+03:00 ???? ?????? : > Yeah. The code: > local line = fmt("%s", "Hi \n all!"); > print line; > > will print the Hi \n all! > :( > > 2017-08-04 16:48 GMT+03:00 anthony kasza : > >> Have you tried using the fmt function in conjunction with the print >> function? >> https://www.bro.org/sphinx/scripts/base/bif/bro.bif.bro.html#id-fmt >> >> -AK >> >> On Aug 4, 2017 6:09 AM, "???? ??????" wrote: >> >>> Hi. I got a trouble with escape sequences in Bro. >>> When i write: >>> print "Next words must be in a new line \n New line" >>> or >>> print "\n" >>> >>> \n will decoded in x0a or \n, but not decoded as a new. >>> >>> So there is a question: how can i make a new line in strings? F.e. i >>> read several strings of data and need to from string like >>> "hi, >>> its a new line >>> and it is too!" >>> >>> but only thing i have is "hi,\nits a new line\nand it is too!" >>> >>> >>> Thanks for answer >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170804/f5823c2f/attachment.html From pmurphy+bro at nrao.edu Fri Aug 4 10:31:04 2017 From: pmurphy+bro at nrao.edu (Patrick P Murphy) Date: Fri, 04 Aug 2017 13:31:04 -0400 Subject: [Bro] Escape sequences in BRO (\n) References: Message-ID: On Fri, 4 Aug 2017 17:04:07 +0300, said: > local line = fmt("%s", "Hi \n all!"); > print line; > will print the Hi \n all! Shouldn't the \n be in the format specifier? Eg., local lines = fmt("%s\n%s\n", "First line", "Second line"); I don't think C or perl or other shell scripts with printf-like capabilities would behave differently: the controls have to be in the format specifier, not the data. - Pat From adh.2234 at gmail.com Fri Aug 4 13:28:56 2017 From: adh.2234 at gmail.com (=?UTF-8?B?0JjQstCw0L0g0KDQsNGC0LrQuNC9?=) Date: Fri, 4 Aug 2017 23:28:56 +0300 Subject: [Bro] Escape sequences in BRO (\n) In-Reply-To: References: Message-ID: Well, the printed line for this sample is: First line\x0aSecond line\x0a 2017-08-04 20:31 GMT+03:00 Patrick P Murphy : > On Fri, 4 Aug 2017 17:04:07 +0300, said: > > > local line = fmt("%s", "Hi \n all!"); > > print line; > > will print the Hi \n all! > > Shouldn't the \n be in the format specifier? Eg., > > local lines = fmt("%s\n%s\n", "First line", "Second line"); > > I don't think C or perl or other shell scripts with printf-like > capabilities would behave differently: the controls have to be in the > format specifier, not the data. > > - Pat > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170804/2f168d60/attachment.html From jazoff at illinois.edu Fri Aug 4 13:44:41 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 4 Aug 2017 20:44:41 +0000 Subject: [Bro] Escape sequences in BRO (\n) In-Reply-To: References: Message-ID: <65836702-745D-43CC-BBAD-5D43452293D0@illinois.edu> Bro escapes strings that are printed by default. You need to open a file(or stdout) in raw mode in order for it to output strings as-is global myfile = open("-"); enable_raw_output(myfile); print myfile, "hello\nworld"; close(myfile); or global myfile = open("-") &raw_output; print myfile, "hello\nworld"; close(my file); -- - Justin Azoff > On Aug 4, 2017, at 4:28 PM, ???? ?????? wrote: > > Well, the printed line for this sample is: > > First line\x0aSecond line\x0a > From adh.2234 at gmail.com Fri Aug 4 20:36:56 2017 From: adh.2234 at gmail.com (=?UTF-8?B?0JjQstCw0L0g0KDQsNGC0LrQuNC9?=) Date: Sat, 5 Aug 2017 06:36:56 +0300 Subject: [Bro] Escape sequences in BRO (\n) In-Reply-To: <65836702-745D-43CC-BBAD-5D43452293D0@illinois.edu> References: <65836702-745D-43CC-BBAD-5D43452293D0@illinois.edu> Message-ID: My thanks! It works. 2017-08-04 23:44 GMT+03:00 Azoff, Justin S : > Bro escapes strings that are printed by default. You need to open a > file(or stdout) in raw mode in order for it to output strings as-is > > global myfile = open("-"); > enable_raw_output(myfile); > print myfile, "hello\nworld"; > close(myfile); > > or > > global myfile = open("-") &raw_output; > print myfile, "hello\nworld"; > close(my file); > > > > -- > - Justin Azoff > > > On Aug 4, 2017, at 4:28 PM, ???? ?????? wrote: > > > > Well, the printed line for this sample is: > > > > First line\x0aSecond line\x0a > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170805/0743ba6f/attachment.html From email4myth at gmail.com Mon Aug 7 00:29:29 2017 From: email4myth at gmail.com (Myth Ren) Date: Mon, 7 Aug 2017 15:29:29 +0800 Subject: [Bro] different file hash between downloaded file by ANALYZER_EXTRACT with original file Message-ID: Hello, everyone . i'm new to bro recently, i'm using FAF(File Analysis Framework) to extract certain type file to disk for further analysis from traffic . but now i have problem which is so difficult to understand: - bro extract file size is one byte bigger than my original file - or bro extract file the right size with my original file, but it's different MD5 value among these files below is my test env, test steps and test result: # my test env bro version: - bro version 2.5-156 OS (32C 64G): - CentOS Linux release 7.3.1611 (Core) CPU model: - Model name: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz - CPU(s): 32 - CPU MHz: 2334.445 NIC: - 03:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network # my test bro scripts ``` event file_sniff(f: fa_file, meta: fa_metadata) { print "file sniff event by Myth"; if ( meta?$mime_type )#&& hook FileExtraction::extract(f, meta) ) { if ( meta$mime_type in mime_to_ext ) { local fext = mime_to_ext[meta$mime_type]; if ( fext == "txt" ) { #print "txt"; if ( f$source != "SMTP" ) { #print "NOT SMTP"; return; } } } else return; #fext = split_string(meta$mime_type, /\//)[1]; local fname = fmt("%s%s-%s.%s", path, f$source, f$id, fext); # file path #print fname; Files::add_analyzer(f, Files::ANALYZER_MD5); Files::add_analyzer(f, Files::ANALYZER_SHA1); Files::add_analyzer(f, Files::ANALYZER_SHA256); Files::add_analyzer(f, Files::ANALYZER_EXTRACT,[$extract_filename=fname]); } } ``` # my test steps 1. generate test file >>> [root at sensor ~]# dd if=/dev/urandom of=test.for.bro.txt bs=1024 count=512 >>> [root at sensor ~]# tar -cvzf test.for.bro.tar.gz test.for.bro.txt 2. original file size and MD5 valud >>> [root at sensor ~]# ls -lt test.for.bro.tar.gz -rw-r--r-- 1 root root 524608 8? 7 13:59 test.for.bro.tar.gz >>> [root at sensor ~]# md5sum test.for.bro.tar.gz 6e755b5c0a7754c7066ca6db5f0f90ba test.for.bro.tar.gz 2. start test web server using Python >>> [root at sensor ~]# python -m SimpleHTTPServer 8998 > ws.log 2>&1 3. start bro >>> [root at sensor myth]# /usr/local/bro/bin/bro -i eno1 -C bro-scripts/tophant.entrypoint.bro > myth.log 2>&1 4. using `ab` do make lots of http request to test file from another machine >>> [root at localhost ~]# ab -n 2000 -c 4 ' http://10.0.81.54:8998/test.for.bro.tar.gz' 5. result ( after all request is done) 5.1 webserver process request count >>> [root at sensor ~]# cat ws.log | grep test.for.bro | wc -l 2000 5.2 bro `file_sniff` event count >>> [root at sensor myth]# cat myth.log | grep "file sniff event by Myth" | wc -l 976 5.3 download file count >>> [root at sensor sensor_files_by_myth]# ls | wc -l 973 5.4 file count with different file size: >>> [root at sensor sensor_files_by_myth]# ls -lt | grep -v 524608 | wc -l 193 5.5 file count with same file size: >>> [root at sensor sensor_files_by_myth]# ls -lt | grep 524608 | wc -l 780 5.6 file count with same MD5 value: >>> [root at sensor sensor_files_by_myth]# ls -lt | awk '{print $NF}' | xargs md5sum | grep 6e755b5c0a7754c7066ca6db5f0f90ba | wc -l 19 5.7 file count with same file size but different MD5 (!!! NOTICE: all is different MD5) >>> [root at sensor sensor_files_by_myth]# ls -lt | grep 524608 | awk '{print $NF}' | xargs md5sum | grep -v 6e755b5c0a7754c7066ca6db5f0f90ba | awk '{print $1}' | sort | uniq -c | wc -l 761 5.8 download file size distribution: >>> [root at sensor sensor_files_by_myth]# ls -lt | awk '{print $5}' | sort -rn | uniq -c * 136 524609 <<<<<<<<<<<<<<< this is one byte bigger than my original test file !!!* * 780 524608* 3 523990 3 522542 8 521094 1 520208 1 519646 2 518198 1 515302 1 513854 1 512968 1 512406 1 510958 1 509510 2 503718 1 502176 1 501384 1 497926 1 490296 1 488808 1 487040 1 486342 1 480550 1 473310 1 467518 1 464622 1 458830 1 453038 1 442902 1 441454 1 396566 1 382408 1 377742 1 358918 1 354574 1 318240 1 283312 1 263350 1 256110 1 250318 1 234952 1 189502 1 164886 1 79454 2 2710 1 Thanks for reading so far, wish someone could help me with this :) Myth -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170807/922175b9/attachment.html From bill.de.ping at gmail.com Tue Aug 8 01:32:38 2017 From: bill.de.ping at gmail.com (william de ping) Date: Tue, 8 Aug 2017 11:32:38 +0300 Subject: [Bro] All cluster instances are stopped yet their process is alive Message-ID: Hi, I am encountering a strange behavior here, I have a cluster (1 manager,1proxy,8 workers) and after a while : - For some reason all of its instances appears as stopped - In the spool directory, I see an empty debug.log, manager folder and NO workers\tmp\proxy folders - Logs are still being written to the manager folder - All of bro's instances are actually still alive I used to have a "broctl cron" task in crontab, but it has been commented out I ran Broctl using root user, and I see that all of bro's processes run as root. any advice on this issue ? Thanks B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170808/d9e434ac/attachment.html From raj at bivio.net Tue Aug 8 12:17:33 2017 From: raj at bivio.net (Raj Srinivasan) Date: Tue, 8 Aug 2017 19:17:33 +0000 Subject: [Bro] Logger crash with Bro 2.5 In-Reply-To: References: Message-ID: [Note to Moderator: Resending with less data since the previous one was too long] Hi, We have been seeing this logger crash with Bro 2.5 running on a PPC based system. I have included the diag output from broctl below. It seems to be related to the rate at which logs are generated - the higher the rate (corresponding to more incoming traffic) the sooner the crash occurs. The CPU on which the logger runs has only about 4GB of memory, out of which approximately 1.5GB is available to the logger. As the logs below show, tc_malloc() does fail to allocate memory a few times, and I am guessing that the disks are not able to keep up with the logging rate. Is the logger capable of recovering from memory allocation failure? If so, does the trace below indicate any other issue? Would appreciate any suggestions and/or insight into the problem. Thanks, Raj [BroControl] > diag logger [logger] Bro 2.5 Linux 2.6.31-45bbv51npc3 Bro plugins: (none found) Core file: core.14433 [New Thread 14454] [New Thread 14455] [New Thread 14713] [New Thread 14714] [New Thread 14717] [New Thread 15060] [New Thread 15061] Core was generated by `/var/tmp/bro/spool/tmp/bro -U .status -p broctl -p broctl -live -p local -p logg'. Program terminated with signal 11, Segmentation fault. #0 0x0f72b598 in *__GI__IO_default_xsputn (f=0xbf8935a8, data=0x1057a43c, n=17) at genops.c:447 in genops.c Thread 8 (Thread 15061): #0 0x1027c89c in threading::formatter::Ascii::Describe(ODesc*, int, threading::Field const* const*, threading::Value**) const () #1 0x1027c7ec in threading::formatter::Ascii::Describe(ODesc*, int, threading::Field const* const*, threading::Value**) const () #2 0x10503a8c in logging::writer::Ascii::DoWrite(int, threading::Field const* const*, threading::Value**) () #3 0x10520a3c in logging::WriterBackend::Write(int, int, threading::Value***) () Backtrace stopped: frame did not save the PC Current language: auto; currently minimal Thread 7 (Thread 15060): #0 0x0fb3591c in __pthread_cond_timedwait (cond=0x12a2e570, mutex=0x12a2e468, abstime=0x4b03fbc8) at pthread_cond_timedwait.c:199 #1 0x1027a000 in threading::MsgThread::Run() () Backtrace stopped: frame did not save the PC Thread 6 (Thread 14717): #0 0x0fb3591c in __pthread_cond_timedwait (cond=0x12a2ef30, mutex=0x12a2edc8, abstime=0x4a83fbc8) at pthread_cond_timedwait.c:199 #1 0x1027a000 in threading::MsgThread::Run() () Backtrace stopped: frame did not save the PC Thread 5 (Thread 14714): #0 0x0fb3591c in __pthread_cond_timedwait (cond=0x12a2f800, mutex=0x12a2f6b0, abstime=0x4a03fbc8) at pthread_cond_timedwait.c:199 #1 0x1027a000 in threading::MsgThread::Run() () Backtrace stopped: frame did not save the PC Thread 4 (Thread 14713): #0 0x0fb3591c in __pthread_cond_timedwait (cond=0x12a2c1a0, mutex=0x12a2c080, abstime=0x4983fbc8) at pthread_cond_timedwait.c:199 #1 0x1027a000 in threading::MsgThread::Run() () Backtrace stopped: frame did not save the PC Thread 3 (Thread 14455): #0 0x0fb3591c in __pthread_cond_timedwait (cond=0x12a2cad0, mutex=0x12a2c998, abstime=0x4903fbc8) at pthread_cond_timedwait.c:199 #1 0x1027a000 in threading::MsgThread::Run() () Backtrace stopped: frame did not save the PC Thread 2 (Thread 14454): #0 0x0fb3591c in __pthread_cond_timedwait (cond=0x12a2d3d0, mutex=0x12a2d298, abstime=0x4883fbc8) at pthread_cond_timedwait.c:199 #1 0x1027a000 in threading::MsgThread::Run() () Backtrace stopped: frame did not save the PC Thread 1 (Thread 14433): ==== reporter.log 1501635254.338084 Reporter::WARNING Your interface is likely receivi ng invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command -line option or toggling the 'ignore_checksums' variable. Alternatively, disabl e checksum offloading by the network adapter to ensure Bro analyzes the actual c hecksums that are transmitted. /usr/local/bro/share/bro/base/misc/find-checksum -offloading.bro, line 54 1501635254.016895 Reporter::WARNING Your interface is likely receivi ng invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command -line option or toggling the 'ignore_checksums' variable. Alternatively, disabl e checksum offloading by the network adapter to ensure Bro analyzes the actual c hecksums that are transmitted. /usr/local/bro/share/bro/base/misc/find-checksum -offloading.bro, line 54 1501635254.486632 Reporter::WARNING Your interface is likely receivi ng invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command -line option or toggling the 'ignore_checksums' variable. Alternatively, disabl e checksum offloading by the network adapter to ensure Bro analyzes the actual c hecksums that are transmitted. /usr/local/bro/share/bro/base/misc/find-checksum -offloading.bro, line 54 1501635254.857459 Reporter::WARNING Your interface is likely receivi ng invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command -line option or toggling the 'ignore_checksums' variable. Alternatively, disabl e checksum offloading by the network adapter to ensure Bro analyzes the actual c hecksums that are transmitted. /usr/local/bro/share/bro/base/misc/find-checksum -offloading.bro, line 54 1501635254.017679 Reporter::WARNING Your interface is likely receivi ng invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command -line option or toggling the 'ignore_checksums' variable. Alternatively, disabl e checksum offloading by the network adapter to ensure Bro analyzes the actual c hecksums that are transmitted. /usr/local/bro/share/bro/base/misc/find-checksum -offloading.bro, line 54 1501635254.365704 Reporter::WARNING Your interface is likely receivi ng invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command -line option or toggling the 'ignore_checksums' variable. Alternatively, disabl e checksum offloading by the network adapter to ensure Bro analyzes the actual c hecksums that are transmitted. /usr/local/bro/share/bro/base/misc/find-checksum -offloading.bro, line 54 1501635254.300063 Reporter::WARNING Your interface is likely receivi ng invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command -line option or toggling the 'ignore_checksums' variable. Alternatively, disabl e checksum offloading by the network adapter to ensure Bro analyzes the actual c hecksums that are transmitted. /usr/local/bro/share/bro/base/misc/find-checksum -offloading.bro, line 54 1501635254.486147 Reporter::WARNING Your interface is likely receivi ng invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command -line option or toggling the 'ignore_checksums' variable. Alternatively, disabl e checksum offloading by the network adapter to ensure Bro analyzes the actual c hecksums that are transmitted. /usr/local/bro/share/bro/base/misc/find-checksum -offloading.bro, line 54 1501635254.686000 Reporter::WARNING Your interface is likely receivi ng invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command -line option or toggling the 'ignore_checksums' variable. Alternatively, disabl e checksum offloading by the network adapter to ensure Bro analyzes the actual c hecksums that are transmitted. /usr/local/bro/share/bro/base/misc/find-checksum -offloading.bro, line 54 1501635254.733590 Reporter::WARNING Your interface is likely receivi ng invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command -line option or toggling the 'ignore_checksums' variable. Alternatively, disabl e checksum offloading by the network adapter to ensure Bro analyzes the actual c hecksums that are transmitted. /usr/local/bro/share/bro/base/misc/find-checksum -offloading.bro, line 54 ==== stderr.log src/central_freelist.cc:322] tcmalloc: allocation failed 16384 out of memory in new. src/central_freelist.cc:322] tcmalloc: allocation failed 16384 out of memory in new. src/central_freelist.cc:322] tcmalloc: allocation failed 16384 out of memory in new. src/central_freelist.cc:322] tcmalloc: allocation failed 16384 out of memory in new. src/central_freelist.cc:322] tcmalloc: allocation failed 16384 /usr/local/bro/share/broctl/scripts/run-bro: line 107: 14433 Segmentation fault (core dumped) nohup "$mybro" "$@" ==== stdout.log max memory size (kbytes, -m) 750000 data seg size (kbytes, -d) 750000 virtual memory (kbytes, -v) 750000 core file size (blocks, -c) unlimited ==== .cmdline -U .status -p broctl -p broctl-live -p local -p logger local.bro broctl base/fra meworks/cluster local-logger.bro broctl/auto ==== .env_vars PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/sbin:/usr /local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/root/bin:/usr/local/bro/bin BROPATH=/var/tmp/bro/spool/installed-scripts-do-not-touch/site::/var/tmp/bro/spo ol/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/s hare/bro/policy:/usr/local/bro/share/bro/site CLUSTER_NODE=logger ==== .status RUNNING [net_run] ==== No prof.log ==== packet_filter.log #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path packet_filter #open 2017-08-01-17-52-24 #fields ts node filter init success #types time string string bool bool -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170808/b743347d/attachment-0001.html From pssunu6 at gmail.com Wed Aug 9 10:28:30 2017 From: pssunu6 at gmail.com (ps sunu) Date: Wed, 9 Aug 2017 22:58:30 +0530 Subject: [Bro] udp module Message-ID: Hi , is udp module is available in bro ? i need this for nfs.bro scrips when i run this nfs.bro script iam getting below message fatal error in ./nfs.bro, line 3: can't find udp regards, Sunu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170809/0058520f/attachment.html From pssunu6 at gmail.com Wed Aug 9 12:06:55 2017 From: pssunu6 at gmail.com (ps sunu) Date: Thu, 10 Aug 2017 00:36:55 +0530 Subject: [Bro] udp module In-Reply-To: References: Message-ID: i fixed that problem and now i am getting below error when i run this error in ./notice.bro, line 112: unknown identifier alarm, at or near "alarm" code alarm fmt("%s (%d time%s)", s, n, n > 1 ? "s" : ""); anythink wrong in this line On Wed, Aug 9, 2017 at 10:58 PM, ps sunu wrote: > Hi , > is udp module is available in bro ? i need this for > nfs.bro scrips when i run this nfs.bro script iam getting below message > > > fatal error in ./nfs.bro, line 3: can't find udp > > > regards, > Sunu > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170810/b15b218d/attachment.html From vikrambasu059 at gmail.com Wed Aug 9 12:35:24 2017 From: vikrambasu059 at gmail.com (Vikram Basu) Date: Thu, 10 Aug 2017 01:05:24 +0530 Subject: [Bro] files.log filename column is blank Message-ID: <598b63fc.5dd1620a.6d818.2ba3@mx.google.com> Hi, I notice there is a filename column in the file.log but it is always empty whether I am doing HTTP transfer or FTP transfer in the network. Why is this the case ? Also when files are transferred over FTP do they show up in the files.log? Because I transferred some files over FTP but even though ftp.log is generated, there is no corresponding entry in files.log What am I doing wrong ? Regards Vikram -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170810/1a47f3e2/attachment.html From johanna at icir.org Wed Aug 9 16:11:51 2017 From: johanna at icir.org (Johanna Amann) Date: Wed, 9 Aug 2017 16:11:51 -0700 Subject: [Bro] files.log filename column is blank In-Reply-To: <598b63fc.5dd1620a.6d818.2ba3@mx.google.com> References: <598b63fc.5dd1620a.6d818.2ba3@mx.google.com> Message-ID: <20170809231151.t5bt7cchfy6qsjyb@user208.sys.ICSI.Berkeley.EDU> Hi Vikram, > I notice there is a filename column in the file.log but it is always empty whether I am doing HTTP transfer or FTP transfer in the network. I am just going to refer back to a previous answer of this question: http://mailman.icsi.berkeley.edu/pipermail/bro/2014-April/006893.html (Short answer: because we don't have a reliable filename). > Also when files are transferred over FTP do they show up in the files.log? Because I transferred some files over FTP but even though ftp.log is generated, there is no corresponding entry in files.log No, I think they don't. If I remember it correctly this is due to the fact that FTP uses separate connections for transferring the data, which (when using clustering) will probably be handled by a different Bro worker than the one handling the original connection; the other worker has no idea that this is an FTP data connection. I hope this helps, Johanna From johanna at icir.org Wed Aug 9 16:17:00 2017 From: johanna at icir.org (Johanna Amann) Date: Wed, 9 Aug 2017 16:17:00 -0700 Subject: [Bro] udp module In-Reply-To: References: Message-ID: <20170809231700.pdqlwuwtzte3ns7s@user208.sys.ICSI.Berkeley.EDU> Hi Sunu, it looks like you found a Bro script that was written for a very old version of Bro (< 2.0). You will probably have to nearly completely re-write that script because of functionality changes; alarm e.g. do not exist anymore (I think they map to notices). Also note that the NFS code in Bro is not regularly tested and there might be bugs lurking in it. Johanna On Thu, Aug 10, 2017 at 12:36:55AM +0530, ps sunu wrote: > i fixed that problem and now i am getting below error when i run this > > error in ./notice.bro, line 112: unknown identifier alarm, at or near > "alarm" > > > code alarm fmt("%s (%d time%s)", s, n, n > 1 ? "s" : ""); anythink > wrong in this line > > On Wed, Aug 9, 2017 at 10:58 PM, ps sunu wrote: > > > Hi , > > is udp module is available in bro ? i need this for > > nfs.bro scrips when i run this nfs.bro script iam getting below message > > > > > > fatal error in ./nfs.bro, line 3: can't find udp > > > > > > regards, > > Sunu > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Wed Aug 9 16:21:05 2017 From: johanna at icir.org (Johanna Amann) Date: Wed, 9 Aug 2017 16:21:05 -0700 Subject: [Bro] Logger crash with Bro 2.5 In-Reply-To: References: Message-ID: <20170809232105.pvcwpqjora6tazzu@user208.sys.ICSI.Berkeley.EDU> On Tue, Aug 08, 2017 at 07:17:33PM +0000, Raj Srinivasan wrote: > We have been seeing this logger crash with Bro 2.5 running on a PPC > based system. I have included the diag output from broctl below. It > seems to be related to the rate at which logs are generated - the higher > the rate (corresponding to more incoming traffic) the sooner the crash > occurs. The CPU on which the logger runs has only about 4GB of memory, > out of which approximately 1.5GB is available to the logger. As the logs > below show, tc_malloc() does fail to allocate memory a few times, and I > am guessing that the disks are not able to keep up with the logging > rate. Yup, that sounds like a reasonable guess from your description. > Is the logger capable of recovering from memory allocation failure? If > so, does the trace below indicate any other issue? Would appreciate any > suggestions and/or insight into the problem. No, it is not - Bro generally is not able to recover from memory allocation failures. I can't really tell anything else from the attached backtrace - they stop a bit too early to be useful. I hope this helps, Johanna From johanna at icir.org Wed Aug 9 16:56:12 2017 From: johanna at icir.org (Johanna Amann) Date: Wed, 9 Aug 2017 16:56:12 -0700 Subject: [Bro] Protocl Analyzer: when to Unref a variable In-Reply-To: <92bf769e-5a0e-ae7e-4639-7a566a6351d8@gmx.com> References: <92bf769e-5a0e-ae7e-4639-7a566a6351d8@gmx.com> Message-ID: <20170809235612.icgau7j2fq3uu7gp@user208.sys.ICSI.Berkeley.EDU> Hi Valerio, the answer to this is more or less that it is a bit complicated. Usually when raising events, the values are "consumed" by the Bro core (so you won't have to call Unref or anything on them). I expect most of the times that you see an Unref directly in an analyzer code is when a data structure is first constructed and then not directly passed to the core, but instead deleted again (e.g. because some precondition was not met). Another case might be when it is replaced with a different structure. When using more complex data structures, things get more complicated and you basically have to resort to reading the code of the data structure you are using, to determine if you have to call Unref afterwards (or you have to use memory leak checks). An example where I think you see explicit unrefs is when using a TableVal; the index is not consumed. I hope this helps a bit, Johanna On Fri, Jul 28, 2017 at 08:47:08AM +0200, Valerio wrote: > By looking at some analyzers (e.g., sip, TCP) I noticed that some here > and there (usually after the content of a variable is assigned to > another structure with Append) a call to Unref() is made. > > Are there any general principles or guidelines regarding when to > explicitly Unref a variable when writing a Bro Analyzer with BinPAC? > > thanks, > Valerio > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From johanna at icir.org Wed Aug 9 16:58:35 2017 From: johanna at icir.org (Johanna Amann) Date: Wed, 9 Aug 2017 16:58:35 -0700 Subject: [Bro] no intel.log | critical-stack intel In-Reply-To: <592228F4D0C8504187F2F76658040CB6E0120C6D@HOT-MAILBOX-02.HOT.NET.IL> References: <592228F4D0C8504187F2F76658040CB6E0120C6D@HOT-MAILBOX-02.HOT.NET.IL> Message-ID: <20170809235835.qjusiijuaxeyoa6i@user208.sys.ICSI.Berkeley.EDU> I can't read their article because it needs a log in. That being said - assuming you follow steps similar to https://www.bro.org/sphinx-git/frameworks/intel.html, especially the redef Intel::read_files part, data should be read in by Bro and the intel.log should get generated. If it does not, check if you have a reporter.log that complains about issues reading the file. Johanna On Wed, Jul 26, 2017 at 07:00:06AM +0000, Izik Birka wrote: > Hi > I installed the critical-stack agent > I pulled the feeds and the master file was created successfully > > But when I trying to test it , and connect to forbidden address , I'm not getting the log in the intel.log like I should > Actually there is no file "intel.log" > > What am I missing ? > > I was following this article https://intel.criticalstack.com/client/0-4-x/usage > > [Enjoy] > > ????? ????? > ??? ???? ????? ???? ?????? ???? > ????? ?????? ???? > 077-7077790 | 053-6064571 > > P ???? ?? ?????? ???? ?????? ???? ?? > > > [Enjoy] > > This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain materials protected by copyright or information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or agreement. > > If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication by error, notify the sender immediately and delete this message immediately. > > Thank you. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From pssunu6 at gmail.com Wed Aug 9 17:54:37 2017 From: pssunu6 at gmail.com (ps sunu) Date: Thu, 10 Aug 2017 06:24:37 +0530 Subject: [Bro] Import old Bro 1.x policy nfs.bro Message-ID: Hi, Any one know bro 1.x version policy nfs.bro have newer version or have alternative script in bro 2.5 ? https://github.com/grigorescu/bro-scripts/blob/master/scripts/todo/needs_review/nfs.bro Regards, Sunu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170810/37c89353/attachment-0001.html From seth at corelight.com Wed Aug 9 19:51:04 2017 From: seth at corelight.com (Seth Hall) Date: Wed, 9 Aug 2017 22:51:04 -0400 Subject: [Bro] different file hash between downloaded file by ANALYZER_EXTRACT with original file In-Reply-To: References: Message-ID: On Mon, Aug 7, 2017 at 3:29 AM, Myth Ren wrote: > - bro extract file size is one byte bigger than my original file > - or bro extract file the right size with my original file, but it's > different MD5 value among these files Ugh, that's not a good behavior. > below is my test env, test steps and test result: Could you capture traffic and replay that with Bro instead of sniffing the interface directly? If you did that you could at least verify that the problem is deterministically replicable and then we could possibly look into the problem with you. I have several thoughts about what the problem could be but they're ultimately fairly long shots and could likely be wrong. .Seth -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From pssunu6 at gmail.com Wed Aug 9 21:22:01 2017 From: pssunu6 at gmail.com (ps sunu) Date: Thu, 10 Aug 2017 09:52:01 +0530 Subject: [Bro] udp.bro and nfs.bro Message-ID: Hi everyone i want to rewrite udp.bro and nfs.bro for bro 2.5 , now its supporting only bro 1.5 , can any one give suggestion how to start this ??? please help to start this Regards, Sunu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170810/c26142ee/attachment.html From pssunu6 at gmail.com Wed Aug 9 22:10:02 2017 From: pssunu6 at gmail.com (ps sunu) Date: Thu, 10 Aug 2017 10:40:02 +0530 Subject: [Bro] Scan::check_scan Message-ID: Hi, any one know hot module and scan module (support bro 1.5) alternative in bro 2.5 ? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170810/8e7569f9/attachment.html From seth at corelight.com Thu Aug 10 04:42:54 2017 From: seth at corelight.com (Seth Hall) Date: Thu, 10 Aug 2017 07:42:54 -0400 Subject: [Bro] Split-ed connection for some UDP traffic? In-Reply-To: References: Message-ID: <598C46BE.5010301@corelight.com> Hi Fatema, I don't see a reply to this message in the mailing list so I'll give it a shot... fatema bannatwala wrote: > 1500927487.398576 CLr9ebnHeAYNOGzei 24.132.204.62 41600 > 128.175.235.216 389 udp - 93.677712 39999 0 S0 F T 0 > D 597 56715 0 0 (empty) > 1500927487.404591 CapBfs1lhI2XFt4gJb 128.175.235.216 389 > 24.132.204.62 41600 udp - 93.672242 1773687 0 S0 T F > 0 D 597 1790403 0 0 (empty) > > Here, in the above case, shouldn't Bro be logging only a single > connection with src: 24.132.204.62 and dest: 128.175.235.216, with > History 'Dd' ? or I might be missing > something important here :) Your traffic isn't being load balanced correctly. You have one worker receiving one flow of the connection and another worker receiving the other flow of the connection. You can tell because of the two different "connections" that have the 4-tuple of ports and ip addresses and you picked up on the "D" instead of "Dd". That just means that traffic was only seen from the originator which we would expect with mismatched load balancing. Are you seeing this sort of behavior with other connections or just this one single odd-ball connection? .Seth -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From valerio.click at gmx.com Thu Aug 10 07:23:37 2017 From: valerio.click at gmx.com (Valerio) Date: Thu, 10 Aug 2017 16:23:37 +0200 Subject: [Bro] Protocl Analyzer: when to Unref a variable In-Reply-To: <20170809235612.icgau7j2fq3uu7gp@user208.sys.ICSI.Berkeley.EDU> References: <92bf769e-5a0e-ae7e-4639-7a566a6351d8@gmx.com> <20170809235612.icgau7j2fq3uu7gp@user208.sys.ICSI.Berkeley.EDU> Message-ID: <91745107-ee65-df53-3239-8d1062d0951d@gmx.com> Hi Johanna, thanks for your reply, it does help a lot! Especially the confirmation that values that are passed to events are "consumed" by Bro core. best, Valerio Il 10/08/2017 01:56, Johanna Amann ha scritto: > Hi Valerio, > > the answer to this is more or less that it is a bit complicated. Usually > when raising events, the values are "consumed" by the Bro core (so you > won't have to call Unref or anything on them). I expect most of the times > that you see an Unref directly in an analyzer code is when a data > structure is first constructed and then not directly passed to the core, > but instead deleted again (e.g. because some precondition was not met). > Another case might be when it is replaced with a different structure. > > When using more complex data structures, things get more complicated and > you basically have to resort to reading the code of the data structure you > are using, to determine if you have to call Unref afterwards (or you have > to use memory leak checks). An example where I think you see explicit > unrefs is when using a TableVal; the index is not consumed. > > I hope this helps a bit, > Johanna > > On Fri, Jul 28, 2017 at 08:47:08AM +0200, Valerio wrote: >> By looking at some analyzers (e.g., sip, TCP) I noticed that some here >> and there (usually after the content of a variable is assigned to >> another structure with Append) a call to Unref() is made. >> >> Are there any general principles or guidelines regarding when to >> explicitly Unref a variable when writing a Bro Analyzer with BinPAC? >> >> thanks, >> Valerio >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> From dnthayer at illinois.edu Thu Aug 10 08:21:51 2017 From: dnthayer at illinois.edu (Daniel Thayer) Date: Thu, 10 Aug 2017 10:21:51 -0500 Subject: [Bro] All cluster instances are stopped yet their process is alive In-Reply-To: References: Message-ID: <9b19bfc3-22db-9d4b-3548-8054d3b22947@illinois.edu> I wonder if you have two different installations of Bro on your machine (such as /usr/local/bro and /usr/local/bro.old, for example). If so, you will need to be careful to use only one of them. Also, be careful to not delete any files in the "spool" directory. Otherwise, broctl might lose track of the bro processes that it started. On 8/8/17 3:32 AM, william de ping wrote: > Hi, > > I am encountering a strange behavior here, I have a cluster (1 > manager,1proxy,8 workers) and after a while : > > * For some reason all of its instances appears as stopped > * In the spool directory, I see an empty debug.log, manager folder and > NO workers\tmp\proxy folders > * Logs are still being written to the manager folder > * All of bro's instances are actually still alive > > I used to have a "broctl cron" task in crontab, but it has been > commented out > > I ran Broctl using root user, and I see that all of bro's processes run > as root. > > > any advice on this issue ? > > > Thanks > > B > From johanna at icir.org Thu Aug 10 15:05:52 2017 From: johanna at icir.org (Johanna Amann) Date: Thu, 10 Aug 2017 15:05:52 -0700 Subject: [Bro] udp.bro and nfs.bro In-Reply-To: References: Message-ID: <20170810220552.b7ssqeuxaximsq6i@user208.sys.ICSI.Berkeley.EDU> Hi Sunu, I would start by looking at the other protocol analyzer scripts in base. If you don't know Bro scripting yet, first playing around with it is probably the best start - we have a number of lessons that introduce you to most of the concepts at try.bro.org. Johanna On Thu, Aug 10, 2017 at 09:52:01AM +0530, ps sunu wrote: > Hi everyone > > i want to rewrite udp.bro and nfs.bro for > bro 2.5 , now its supporting only bro 1.5 , can any one give suggestion how > to start this ??? please help to start this > > > Regards, > Sunu > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Thu Aug 10 17:20:36 2017 From: johanna at icir.org (Johanna Amann) Date: Thu, 10 Aug 2017 17:20:36 -0700 Subject: [Bro] Scan::check_scan In-Reply-To: References: Message-ID: <20170811002036.zmziele5zdyomkff@user208.sys.ICSI.Berkeley.EDU> I think https://github.com/initconf/scan-NG/blob/master/scripts/old-scan.bro is a port of the old scan module. May I ask why you need and can't use the one that currently ships with Bro? Johanna On Thu, Aug 10, 2017 at 10:40:02AM +0530, ps sunu wrote: > Hi, > any one know hot module and scan module (support bro 1.5) > alternative in bro 2.5 ? > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From pssunu6 at gmail.com Thu Aug 10 17:25:43 2017 From: pssunu6 at gmail.com (ps sunu) Date: Fri, 11 Aug 2017 05:55:43 +0530 Subject: [Bro] Scan::check_scan In-Reply-To: <20170811002036.zmziele5zdyomkff@user208.sys.ICSI.Berkeley.EDU> References: <20170811002036.zmziele5zdyomkff@user208.sys.ICSI.Berkeley.EDU> Message-ID: I need to run old NFS.bro script into bro 2.5 this script calling UDP script udp-common and its calling scan.bro,hot.bro https://github.com/grigorescu/bro-scripts/blob/master/scripts/todo/needs_review/nfs.bro On Aug 11, 2017 5:50 AM, "Johanna Amann" wrote: > I think > https://github.com/initconf/scan-NG/blob/master/scripts/old-scan.bro is a > port of the old scan module. > > May I ask why you need and can't use the one that currently ships with > Bro? > > Johanna > > On Thu, Aug 10, 2017 at 10:40:02AM +0530, ps sunu wrote: > > Hi, > > any one know hot module and scan module (support bro 1.5) > > alternative in bro 2.5 ? > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170811/080c6f87/attachment.html From johanna at icir.org Thu Aug 10 17:37:54 2017 From: johanna at icir.org (Johanna Amann) Date: Thu, 10 Aug 2017 17:37:54 -0700 Subject: [Bro] Scan::check_scan In-Reply-To: References: <20170811002036.zmziele5zdyomkff@user208.sys.ICSI.Berkeley.EDU> Message-ID: Ok, in that case - as I said before, you should probably just do a complete re-write of the script; you definitely cannot just include random old bro script files and hope that they work :) Johanna On 10 Aug 2017, at 17:25, ps sunu wrote: > I need to run old NFS.bro script into bro 2.5 this script calling UDP > script udp-common and its calling scan.bro,hot.bro > > https://github.com/grigorescu/bro-scripts/blob/master/scripts/todo/needs_review/nfs.bro > > On Aug 11, 2017 5:50 AM, "Johanna Amann" wrote: > >> I think >> https://github.com/initconf/scan-NG/blob/master/scripts/old-scan.bro >> is a >> port of the old scan module. >> >> May I ask why you need and can't use the one that currently ships >> with >> Bro? >> >> Johanna >> >> On Thu, Aug 10, 2017 at 10:40:02AM +0530, ps sunu wrote: >>> Hi, >>> any one know hot module and scan module (support bro >>> 1.5) >>> alternative in bro 2.5 ? >> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> From reswob10 at gmail.com Thu Aug 10 19:34:21 2017 From: reswob10 at gmail.com (craig bowser) Date: Thu, 10 Aug 2017 22:34:21 -0400 Subject: [Bro] Email Notice attempt #2 Message-ID: OK, I've been continuing to work at this and I found Scott Runnell's most excellent blog posts and I've been following part #3. http://ryesecurity.blogspot.com/2012/05/learning-bro-scripting-language.html Now I know that bro has updated a lot since then, but I think I've got the syntax right. However, while the code works and I get one notice in my notice.log, I'm not getting an emailed alert. What am I still missing? Thanks. ------------------------------------------- module HTTP; export { redef enum Notice::Type += { ## Generated if a site is detected using Basic Access Authentication HTTP::Basic_Auth_Server }; } redef Notice::mail_dest = "reswob10 at gmail.com"; redef Notice::emailed_types += { HTTP::Basic_Auth_Server }; hook Notice::policy(n: Notice::Info) { if ( n$note == HTTP::Basic_Auth_Server ) add n$actions[Notice::ACTION_EMAIL]; } event http_header(c: connection, is_orig: bool, name: string, value: string) { if (/AUTHORIZATION/ in name && /Basic/ in value) { NOTICE([$note=HTTP::Basic_Auth_Server, $identifier=cat(c$id$resp_h, c$id$resp_p), $suppress_for=1day, $conn=c]); } } Craig L Bowser ____________________________ This email is measured by size. Bits and bytes may have settled during transport. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170810/4b741fa7/attachment.html From pssunu6 at gmail.com Thu Aug 10 23:57:39 2017 From: pssunu6 at gmail.com (ps sunu) Date: Fri, 11 Aug 2017 12:27:39 +0530 Subject: [Bro] Scan::check_scan In-Reply-To: References: <20170811002036.zmziele5zdyomkff@user208.sys.ICSI.Berkeley.EDU> Message-ID: Hi , I Started doing nfs ANALYZER .bro script in bro 2.5 script , i the first script i am getting below error *const ports = {111/tcp, 111/udp, 747/udp, 759/tcp, 762/udp, 764/tcp,* *2049/udp};* *redef likely_server_ports += {ports};* *event bro_init() &priority=5* * {* * Analyzer::register_for_ports(Analyzer::ANALYZER_NFS, ports);* * }* *event nfs_proc_getattr(c: connection, info: NFS3::info_t, fh: string,**attrs: NFS3::fattr_t){* * print c;* *}* and am getting below error 944207397.280000 internal error: unknown analyzer name RPC; mismatch with tag analyzer::Component? Aborted (core dumped) Regards, Sunu On Aug 11, 2017 6:07 AM, "Johanna Amann" wrote: > Ok, in that case - as I said before, you should probably just do a > complete re-write of the script; you definitely cannot just include random > old bro script files and hope that they work :) > > Johanna > > On 10 Aug 2017, at 17:25, ps sunu wrote: > > I need to run old NFS.bro script into bro 2.5 this script calling UDP >> script udp-common and its calling scan.bro,hot.bro >> >> https://github.com/grigorescu/bro-scripts/blob/master/script >> s/todo/needs_review/nfs.bro >> >> On Aug 11, 2017 5:50 AM, "Johanna Amann" wrote: >> >> I think >>> https://github.com/initconf/scan-NG/blob/master/scripts/old-scan.bro is >>> a >>> port of the old scan module. >>> >>> May I ask why you need and can't use the one that currently ships with >>> Bro? >>> >>> Johanna >>> >>> On Thu, Aug 10, 2017 at 10:40:02AM +0530, ps sunu wrote: >>> >>>> Hi, >>>> any one know hot module and scan module (support bro 1.5) >>>> alternative in bro 2.5 ? >>>> >>> >>> _______________________________________________ >>>> Bro mailing list >>>> bro at bro-ids.org >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>>> >>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170811/fd5c3f04/attachment-0001.html From pssunu6 at gmail.com Thu Aug 10 23:58:33 2017 From: pssunu6 at gmail.com (ps sunu) Date: Fri, 11 Aug 2017 12:28:33 +0530 Subject: [Bro] RPC; mismatch with tag analyzer::Component error in NFS ANALYZER_NFS Message-ID: Hi , I Started doing nfs ANALYZER .bro script in bro 2.5 script , i the first script i am getting below error *const ports = {111/tcp, 111/udp, 747/udp, 759/tcp, 762/udp, 764/tcp,* *2049/udp};* *redef likely_server_ports += {ports};* *event bro_init() &priority=5* * {* * Analyzer::register_for_ports(Analyzer::ANALYZER_NFS, ports);* * }* *event nfs_proc_getattr(c: connection, info: NFS3::info_t, fh: string,**attrs: NFS3::fattr_t){* * print c;* *}* and am getting below error 944207397.280000 internal error: unknown analyzer name RPC; mismatch with tag analyzer::Component? Aborted (core dumped) Regards, Sunu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170811/1cfa8a0f/attachment.html From email4myth at gmail.com Fri Aug 11 00:00:50 2017 From: email4myth at gmail.com (Myth Ren) Date: Fri, 11 Aug 2017 15:00:50 +0800 Subject: [Bro] different file hash between downloaded file by ANALYZER_EXTRACT with original file In-Reply-To: References: Message-ID: i had tests with pcaps, and there is no problem with that. but when i listen on interface directly, weird problem happened, and it happened most times . I opened a issue at https://bro-tracker.atlassian.net/projects/BIT/issues/BIT-1832 . and i upload three files to that issue: 1. test4faf.bro - this is the bro script i use for test 2. test4faf.tar.gz - this is the file i use http to download, this is generated with command `dd if=/dev/urandom of=test4faf.dat bs=1024 count=128 && tar -cvzf test4faf.tar.gz test4faf.dat && rm -f test4faf.dat` 3. test4faf.pcap - this is generated with tcpdump. if i test with this pcap, no problem happened, everything is all right. everyone who has interesting with this problem could do some test with that bro script, but remember to sniffing traffic directly from interface. 2017-08-10 10:51 GMT+08:00 Seth Hall : > On Mon, Aug 7, 2017 at 3:29 AM, Myth Ren wrote: > > - bro extract file size is one byte bigger than my original file > > - or bro extract file the right size with my original file, but it's > > different MD5 value among these files > > Ugh, that's not a good behavior. > > > below is my test env, test steps and test result: > > Could you capture traffic and replay that with Bro instead of sniffing > the interface directly? If you did that you could at least verify > that the problem is deterministically replicable and then we could > possibly look into the problem with you. I have several thoughts > about what the problem could be but they're ultimately fairly long > shots and could likely be wrong. > > .Seth > > -- > Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170811/2958ab8d/attachment.html From dopheide at gmail.com Fri Aug 11 09:24:05 2017 From: dopheide at gmail.com (Mike Dopheide) Date: Fri, 11 Aug 2017 11:24:05 -0500 Subject: [Bro] Potential 'Coding Day' after BroCon Message-ID: Would other people be interested in hanging out an extra day after BroCon to collaborate, write scripts, etc? Obviously, this is totally unofficial. I have a tendency to get a bunch of fun ideas at a conference and it would be great to set aside some time to work on them immediately while the ideas are fresh and motivation is high. It's also easier to do when the time is blocked off on my calendar as being part of a conference so 'normal work' doesn't get in the way. :) I realize if people have already booked travel it may be difficult to adjust, but I'd like to gauge interest. -Dop -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170811/ca87fce4/attachment.html From shirkdog.bsd at gmail.com Fri Aug 11 09:46:59 2017 From: shirkdog.bsd at gmail.com (Michael Shirk) Date: Fri, 11 Aug 2017 12:46:59 -0400 Subject: [Bro] Potential 'Coding Day' after BroCon In-Reply-To: References: Message-ID: After the conference possibly, at least a get together would be nice. -- Michael Shirk Daemon Security, Inc. https://www.daemon-security.com On Aug 11, 2017 12:33 PM, "Mike Dopheide" wrote: > Would other people be interested in hanging out an extra day after BroCon > to collaborate, write scripts, etc? Obviously, this is totally unofficial. > > I have a tendency to get a bunch of fun ideas at a conference and it would > be great to set aside some time to work on them immediately while the ideas > are fresh and motivation is high. It's also easier to do when the time is > blocked off on my calendar as being part of a conference so 'normal work' > doesn't get in the way. :) > > I realize if people have already booked travel it may be difficult to > adjust, but I'd like to gauge interest. > > -Dop > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170811/7652ce14/attachment.html From matt.clemons at gmail.com Fri Aug 11 09:48:29 2017 From: matt.clemons at gmail.com (Matt Clemons) Date: Fri, 11 Aug 2017 11:48:29 -0500 Subject: [Bro] Potential 'Coding Day' after BroCon In-Reply-To: References: Message-ID: <9c66184f-88fc-4ba6-7a90-3449616289eb@gmail.com> The second half of Thursday would be great for that. //Matt On 08/11/2017 11:24 AM, Mike Dopheide wrote: > Would other people be interested in hanging out an extra day after > BroCon to collaborate, write scripts, etc? Obviously, this is totally > unofficial. > > I have a tendency to get a bunch of fun ideas at a conference and it > would be great to set aside some time to work on them immediately while > the ideas are fresh and motivation is high. It's also easier to do when > the time is blocked off on my calendar as being part of a conference so > 'normal work' doesn't get in the way. :) > > I realize if people have already booked travel it may be difficult to > adjust, but I'd like to gauge interest. > > -Dop > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Regards, Matt Clemons 816-200-0789 GPG: http://gpg.mattclemons.com From zeolla at gmail.com Fri Aug 11 10:03:27 2017 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Fri, 11 Aug 2017 17:03:27 +0000 Subject: [Bro] Potential 'Coding Day' after BroCon In-Reply-To: References: Message-ID: I fly out early the last night, but I'm interested in anything I can make work. Jon On Fri, Aug 11, 2017 at 1:01 PM Michael Shirk wrote: > After the conference possibly, at least a get together would be nice. > > -- > Michael Shirk > Daemon Security, Inc. > https://www.daemon-security.com > > On Aug 11, 2017 12:33 PM, "Mike Dopheide" wrote: > >> Would other people be interested in hanging out an extra day after BroCon >> to collaborate, write scripts, etc? Obviously, this is totally unofficial. >> >> I have a tendency to get a bunch of fun ideas at a conference and it >> would be great to set aside some time to work on them immediately while the >> ideas are fresh and motivation is high. It's also easier to do when the >> time is blocked off on my calendar as being part of a conference so 'normal >> work' doesn't get in the way. :) >> >> I realize if people have already booked travel it may be difficult to >> adjust, but I'd like to gauge interest. >> >> -Dop >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170811/776f8017/attachment-0001.html From fatema.bannatwala at gmail.com Fri Aug 11 10:10:16 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Fri, 11 Aug 2017 13:10:16 -0400 Subject: [Bro] Potential 'Coding Day' after BroCon Message-ID: That's a really good thought, I would love too, but already booked my travel months ago, and will be flying out on the Thursday afternoon. I wish I would have known before. :/ Thanks, Fatema. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170811/2bfd8732/attachment.html From BLMILLER at comerica.com Fri Aug 11 10:26:58 2017 From: BLMILLER at comerica.com (Miller, Brad L) Date: Fri, 11 Aug 2017 17:26:58 +0000 Subject: [Bro] Potential 'Coding Day' after BroCon In-Reply-To: References: Message-ID: Unfortunately I wouldn?t be able to participate because I have already scheduled travel around the assumption that Thursday was a half-travel day. From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Mike Dopheide Sent: Friday, August 11, 2017 12:24 PM To: bro at bro.org Subject: [Bro] Potential 'Coding Day' after BroCon Would other people be interested in hanging out an extra day after BroCon to collaborate, write scripts, etc? Obviously, this is totally unofficial. I have a tendency to get a bunch of fun ideas at a conference and it would be great to set aside some time to work on them immediately while the ideas are fresh and motivation is high. It's also easier to do when the time is blocked off on my calendar as being part of a conference so 'normal work' doesn't get in the way. :) I realize if people have already booked travel it may be difficult to adjust, but I'd like to gauge interest. -Dop Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com to submit a secure form using any of the ?Contact Us? forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170811/683367a4/attachment.html From pssunu6 at gmail.com Fri Aug 11 10:32:10 2017 From: pssunu6 at gmail.com (ps sunu) Date: Fri, 11 Aug 2017 23:02:10 +0530 Subject: [Bro] NFS.CC code error Message-ID: Hi, After doing your NFS.CC fix NFS.bro is going to next steps after that i am getting error bro-2.5.1/src/analyzer/protocol/rpc/NFS.cc line no 649 : RPC_Analyzer("RPC", conn, new NFS_Interp(this)) to : RPC_Analyzer("NFS", conn, new NFS_Interp(this)) binu at binu-HP-Pavilion-dv7-Notebook-PC /usr/local/bro/bin $ sudo ./bro -C -r /home/binu/Downloads/nfsv3.pcap /home/binu/nfs.bro [id=[orig_h=139.25.22.2, orig_p=1022/udp, resp_h=139.25.22.102, resp_p=2049/udp], orig=[size=128, state=1, num_pkts=1, num_bytes_ip=156, flow_label=0, l2_addr=00:c0:95:e0:19:be], resp=[size=112, state=1, num_pkts=0, num_bytes_ip=0, flow_label=0, l2_addr=00:c0:95:f8:4d:d3], start_time=944207397.4, duration=0.0, service={ }, history=Dd, uid=CcWDYV1vmfWIGfA3j7, tunnel=, vlan=, inner_vlan=, dpd=, conn=, extract_orig=F, extract_resp=F, thresholds=, dce_rpc=, dce_rpc_state=, dce_rpc_backing=, dhcp=, dnp3=, dns=, dns_state=, ftp=, ftp_data_reuse=F, ssl=, http=, http_state=, irc=, krb=, modbus=, mysql=, ntlm=, radius=, rdp=, rfb=, sip=, sip_state=, snmp=, smtp=, smtp_state=, socks=, ssh=, syslog=] 944207397.400000 internal error: bad reference count [2] Aborted Regards, Binu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170811/e4529379/attachment.html From seth at corelight.com Fri Aug 11 20:24:18 2017 From: seth at corelight.com (Seth Hall) Date: Fri, 11 Aug 2017 23:24:18 -0400 Subject: [Bro] different file hash between downloaded file by ANALYZER_EXTRACT with original file In-Reply-To: References: Message-ID: <598E74E2.6020802@corelight.com> Myth Ren wrote: > everyone who has interesting with this problem could do some test with > that bro script, but remember to sniffing traffic directly from interface. If you are only seeing the problem when sniffing from an interface, it's likely that the problem is actually that you are dropping packets. When you sniff from an interface, what is your traffic rate that is being monitored? .Seth -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From reswob10 at gmail.com Sat Aug 12 10:11:46 2017 From: reswob10 at gmail.com (craig bowser) Date: Sat, 12 Aug 2017 13:11:46 -0400 Subject: [Bro] Email Notice attempt #2 In-Reply-To: References: Message-ID: OK, further refinement: I've been going over the documentation for notices and raising alerts and googling and I think I've got the right code syntax, but I still am not getting emailed alerts. I configured the following line in /usr/local/bro/share/bro/ base/frameworks/notice/main.bro const mail_dest = "reswob10 at gmail.com" &redef; And here is my script: module HTTP; export { redef enum Notice::Type += { ## Generated if a site is detected using Basic Access Authentication HTTP::Basic_Auth_Server }; } redef Notice::mail_dest = "reswob10 at gmail.com"; hook Notice::policy(n: Notice::Info) { if (n$note == HTTP::Basic_Auth_Server) print n$note; add n$actions[Notice::ACTION_EMAIL]; } event http_header(c: connection, is_orig: bool, name: string, value: string) { if (/AUTHORIZATION/ in name && /Basic/ in value) { # local parts: string_array; # parts = split1(decode_base64(sub_bytes(value, 7, |value|)), /:/); # if (|parts| == 2) NOTICE([$note=HTTP::Basic_Auth_Server, $identifier=cat(c$id$resp_h, c$id$resp_p), $suppress_for=1day, $conn=c ]); } } When I run the script against a local pcap with HTTP basic auth traffic, I get the printed line: root at raspberrypi:/home/pi# bro -C -r http-bro.pcap http-auth-notice.bro HTTP::Basic_Auth_Server But I never get the emailed alert. I AM getting hourly Connection Summary emails so I know bro can send email to my gmail address. Would the alerts be included inside those emails? Could you please provide any tips/suggestions/corrections/rebukes for what I'm doing? Are there other configurations I missed? Is there a general log for bro that can show an error if it's trying to send email but it can't? Thanks. Craig L Bowser ____________________________ This email is measured by size. Bits and bytes may have settled during transport. On Thu, Aug 10, 2017 at 10:34 PM, craig bowser wrote: > > OK, I've been continuing to work at this and I found Scott Runnell's most > excellent blog posts and I've been following part #3. > > http://ryesecurity.blogspot.com/2012/05/learning-bro- > scripting-language.html > > Now I know that bro has updated a lot since then, but I think I've got the > syntax right. However, while the code works and I get one notice in my > notice.log, I'm not getting an emailed alert. > > What am I still missing? > > Thanks. > > ------------------------------------------- > > > > module HTTP; > > export { > redef enum Notice::Type += { > ## Generated if a site is detected using Basic Access Authentication > HTTP::Basic_Auth_Server > }; > } > > redef Notice::mail_dest = "reswob10 at gmail.com"; > redef Notice::emailed_types += { HTTP::Basic_Auth_Server }; > > > hook Notice::policy(n: Notice::Info) > { > if ( n$note == HTTP::Basic_Auth_Server ) > add n$actions[Notice::ACTION_EMAIL]; > } > > event http_header(c: connection, is_orig: bool, name: string, value: > string) > { > if (/AUTHORIZATION/ in name && /Basic/ in value) > { > NOTICE([$note=HTTP::Basic_Auth_Server, > $identifier=cat(c$id$resp_h, c$id$resp_p), > $suppress_for=1day, > $conn=c]); > } > } > > > > Craig L Bowser > ____________________________ > > This email is measured by size. Bits and bytes may have settled during > transport. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170812/6fdc5188/attachment.html From josh.guild at morphick.com Sat Aug 12 11:58:54 2017 From: josh.guild at morphick.com (Josh Guild) Date: Sat, 12 Aug 2017 18:58:54 +0000 Subject: [Bro] Reading encrypted pcap with Bro Message-ID: Hi all, Hoping to find some more uplifting answers here than I found with my Google searches. I have an encrypted pcap and the key but there doesn't seem to be a way to save of the plaintext pcap with tshark. Where Bro comes in - I need to carve some files out that are chunked as octet streams and would really rather not have to write a tshark script for this. However Bro needs the decrypted pcap to carve for me :( Any assistance or points in the right direction would be awesome, thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170812/f50ecc27/attachment.html From mabuchan at gmail.com Sat Aug 12 15:17:08 2017 From: mabuchan at gmail.com (Mark Buchanan) Date: Sat, 12 Aug 2017 17:17:08 -0500 Subject: [Bro] Reading encrypted pcap with Bro In-Reply-To: References: Message-ID: Check out sslviewd, it can do decrypt of traffic (on the fly). You may be able to use that to either generate clear text captures or replay the encrypted dump through it into a Bro instance listening to the output. On another note, Wireshark has some capacity to carve files out, within it, while I know I'd like to use Bro, if it's a one shot deal, that may be an easier method. -- Mark Buchanan > On Aug 12, 2017, at 13:58, Josh Guild wrote: > > Hi all, > > Hoping to find some more uplifting answers here than I found with my Google searches. I have an encrypted pcap and the key but there doesn't seem to be a way to save of the plaintext pcap with tshark. > > Where Bro comes in - I need to carve some files out that are chunked as octet streams and would really rather not have to write a tshark script for this. > > However Bro needs the decrypted pcap to carve for me :( > > Any assistance or points in the right direction would be awesome, thanks! > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From josh.guild at morphick.com Sat Aug 12 15:20:39 2017 From: josh.guild at morphick.com (Josh Guild) Date: Sat, 12 Aug 2017 22:20:39 +0000 Subject: [Bro] Reading encrypted pcap with Bro In-Reply-To: References: Message-ID: Awesome, I'll give that a shot! RE: the replay, is there something that can read that out and reply? I was thinking of just a trying this with tshark but hadn't done research yet. I tried the Export Objects within Wireshark but these files weren't grabbed through a normal GET, it was pushed out in a chunked format. I'm hoping Bro can reassemble and carve for me :) On Sat, Aug 12, 2017, 18:17 Mark Buchanan wrote: > Check out sslviewd, it can do decrypt of traffic (on the fly). You may > be able to use that to either generate clear text captures or replay the > encrypted dump through it into a Bro instance listening to the output. > > On another note, Wireshark has some capacity to carve files out, within > it, while I know I'd like to use Bro, if it's a one shot deal, that may be > an easier method. > > -- > Mark Buchanan > > > On Aug 12, 2017, at 13:58, Josh Guild wrote: > > > > Hi all, > > > > Hoping to find some more uplifting answers here than I found with my > Google searches. I have an encrypted pcap and the key but there doesn't > seem to be a way to save of the plaintext pcap with tshark. > > > > Where Bro comes in - I need to carve some files out that are chunked as > octet streams and would really rather not have to write a tshark script for > this. > > > > However Bro needs the decrypted pcap to carve for me :( > > > > Any assistance or points in the right direction would be awesome, thanks! > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170812/81fffc99/attachment.html From mabuchan at gmail.com Sat Aug 12 15:23:04 2017 From: mabuchan at gmail.com (Mark Buchanan) Date: Sat, 12 Aug 2017 17:23:04 -0500 Subject: [Bro] Reading encrypted pcap with Bro In-Reply-To: References: Message-ID: Ack - sorry - viewssld - I got the name backwards and google finds all sorts of other things when you try sslviewd. Mark On Sat, Aug 12, 2017 at 5:20 PM, Josh Guild wrote: > Awesome, I'll give that a shot! RE: the replay, is there something that > can read that out and reply? I was thinking of just a trying this with > tshark but hadn't done research yet. > > I tried the Export Objects within Wireshark but these files weren't > grabbed through a normal GET, it was pushed out in a chunked format. > > I'm hoping Bro can reassemble and carve for me :) > > On Sat, Aug 12, 2017, 18:17 Mark Buchanan wrote: > >> Check out sslviewd, it can do decrypt of traffic (on the fly). You may >> be able to use that to either generate clear text captures or replay the >> encrypted dump through it into a Bro instance listening to the output. >> >> On another note, Wireshark has some capacity to carve files out, within >> it, while I know I'd like to use Bro, if it's a one shot deal, that may be >> an easier method. >> >> -- >> Mark Buchanan >> >> > On Aug 12, 2017, at 13:58, Josh Guild wrote: >> > >> > Hi all, >> > >> > Hoping to find some more uplifting answers here than I found with my >> Google searches. I have an encrypted pcap and the key but there doesn't >> seem to be a way to save of the plaintext pcap with tshark. >> > >> > Where Bro comes in - I need to carve some files out that are chunked as >> octet streams and would really rather not have to write a tshark script for >> this. >> > >> > However Bro needs the decrypted pcap to carve for me :( >> > >> > Any assistance or points in the right direction would be awesome, >> thanks! >> > _______________________________________________ >> > Bro mailing list >> > bro at bro-ids.org >> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > -- Mark Buchanan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170812/1e4e6dcf/attachment.html From josh.guild at morphick.com Sat Aug 12 15:24:27 2017 From: josh.guild at morphick.com (Josh Guild) Date: Sat, 12 Aug 2017 22:24:27 +0000 Subject: [Bro] Reading encrypted pcap with Bro In-Reply-To: References: Message-ID: Ha. No worries, I'll take a look tonight! On Sat, Aug 12, 2017, 18:23 Mark Buchanan wrote: > Ack - sorry - viewssld - I got the name backwards and google finds all > sorts of other things when you try sslviewd. > > Mark > > On Sat, Aug 12, 2017 at 5:20 PM, Josh Guild > wrote: > >> Awesome, I'll give that a shot! RE: the replay, is there something that >> can read that out and reply? I was thinking of just a trying this with >> tshark but hadn't done research yet. >> >> I tried the Export Objects within Wireshark but these files weren't >> grabbed through a normal GET, it was pushed out in a chunked format. >> >> I'm hoping Bro can reassemble and carve for me :) >> >> On Sat, Aug 12, 2017, 18:17 Mark Buchanan wrote: >> >>> Check out sslviewd, it can do decrypt of traffic (on the fly). You may >>> be able to use that to either generate clear text captures or replay the >>> encrypted dump through it into a Bro instance listening to the output. >>> >>> On another note, Wireshark has some capacity to carve files out, within >>> it, while I know I'd like to use Bro, if it's a one shot deal, that may be >>> an easier method. >>> >>> -- >>> Mark Buchanan >>> >>> > On Aug 12, 2017, at 13:58, Josh Guild wrote: >>> > >>> > Hi all, >>> > >>> > Hoping to find some more uplifting answers here than I found with my >>> Google searches. I have an encrypted pcap and the key but there doesn't >>> seem to be a way to save of the plaintext pcap with tshark. >>> > >>> > Where Bro comes in - I need to carve some files out that are chunked >>> as octet streams and would really rather not have to write a tshark script >>> for this. >>> > >>> > However Bro needs the decrypted pcap to carve for me :( >>> > >>> > Any assistance or points in the right direction would be awesome, >>> thanks! >>> > _______________________________________________ >>> > Bro mailing list >>> > bro at bro-ids.org >>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >> > > > -- > Mark Buchanan > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170812/03227337/attachment.html From oelnaggar04 at gmail.com Sat Aug 12 18:07:02 2017 From: oelnaggar04 at gmail.com (Osama Elnaggar) Date: Sun, 13 Aug 2017 03:07:02 +0200 Subject: [Bro] Reading encrypted pcap with Bro In-Reply-To: References: Message-ID: Just a heads up on viewssld: it only supports old cipher suites. I was looking into using it with Bro to perform SSL decryption and then ran into this paper (published a few months ago) where the author tried to do something similar with Snort - https://www.sans.org/reading-room/whitepapers/detection/snort-ssl-tls-inspection-37735 Here is what the paper's author wrote about viewssld: "Viewssld was only able to decrypt one cipher suite 'TLS_RSA_WITH_RC4_128_MD5'. After checking libdssl-master (Viewssld dependency library), only the cipher suites below were listed in the source code and are currently supported by Viewssld. TLS - AES_128_CBC,SHA1 - AES_256_CBC,SHA1 SSL2 - RC4,MD5 - RC4,MD5 - RC2,MD5 - RC2,MD5 - IDEA,MD5 - DES,MD5 - SN_DES_EDE3_CBC,MD5 SSL3 - NULL,MD5 - NULL,SHA1 - RC4,MD5 - RC4,MD5 - RC4,SHA1 - RC2,MD5 - IDEA,SHA1 - DES,SHA1 - DES,SHA1 - DES3,SHA1 Unfortunately, this tool supports old cipher suites that are insecure and rarely used by servers today. However, the capability here is limited to the tool and what it supports. It is possible to enhance the open source tool?s cipher suite support, but that would take development effort." -- Osama Elnaggar On August 13, 2017 at 8:54:44 AM, Mark Buchanan (mabuchan at gmail.com) wrote: Ack - sorry - viewssld - I got the name backwards and google finds all sorts of other things when you try sslviewd. Mark On Sat, Aug 12, 2017 at 5:20 PM, Josh Guild wrote: > Awesome, I'll give that a shot! RE: the replay, is there something that > can read that out and reply? I was thinking of just a trying this with > tshark but hadn't done research yet. > > I tried the Export Objects within Wireshark but these files weren't > grabbed through a normal GET, it was pushed out in a chunked format. > > I'm hoping Bro can reassemble and carve for me :) > > On Sat, Aug 12, 2017, 18:17 Mark Buchanan wrote: > >> Check out sslviewd, it can do decrypt of traffic (on the fly). You may >> be able to use that to either generate clear text captures or replay the >> encrypted dump through it into a Bro instance listening to the output. >> >> On another note, Wireshark has some capacity to carve files out, within >> it, while I know I'd like to use Bro, if it's a one shot deal, that may be >> an easier method. >> >> -- >> Mark Buchanan >> >> > On Aug 12, 2017, at 13:58, Josh Guild wrote: >> > >> > Hi all, >> > >> > Hoping to find some more uplifting answers here than I found with my >> Google searches. I have an encrypted pcap and the key but there doesn't >> seem to be a way to save of the plaintext pcap with tshark. >> > >> > Where Bro comes in - I need to carve some files out that are chunked as >> octet streams and would really rather not have to write a tshark script for >> this. >> > >> > However Bro needs the decrypted pcap to carve for me :( >> > >> > Any assistance or points in the right direction would be awesome, >> thanks! >> > _______________________________________________ >> > Bro mailing list >> > bro at bro-ids.org >> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > -- Mark Buchanan _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170813/65f69f0f/attachment-0001.html From pssunu6 at gmail.com Sun Aug 13 21:34:53 2017 From: pssunu6 at gmail.com (ps sunu) Date: Mon, 14 Aug 2017 10:04:53 +0530 Subject: [Bro] NFS.CC code error In-Reply-To: References: Message-ID: i am seeing below message in dmesg bro[1454]: segfault at 7f27ebb17bf8 ip 00007f27ebb17bf8 sp 00007ffc4e9d2ae8 error 15 in libc-2.24.so[7f27ebb17000+2000] On Fri, Aug 11, 2017 at 11:02 PM, ps sunu wrote: > Hi, > After doing your NFS.CC fix NFS.bro is going to > next steps after that i am getting error > > > bro-2.5.1/src/analyzer/protocol/rpc/NFS.cc line no 649 > : RPC_Analyzer("RPC", conn, new NFS_Interp(this)) to : > RPC_Analyzer("NFS", conn, new NFS_Interp(this)) > > > binu at binu-HP-Pavilion-dv7-Notebook-PC /usr/local/bro/bin $ sudo ./bro -C > -r /home/binu/Downloads/nfsv3.pcap /home/binu/nfs.bro > [id=[orig_h=139.25.22.2, orig_p=1022/udp, resp_h=139.25.22.102, > resp_p=2049/udp], orig=[size=128, state=1, num_pkts=1, num_bytes_ip=156, > flow_label=0, l2_addr=00:c0:95:e0:19:be], resp=[size=112, state=1, > num_pkts=0, num_bytes_ip=0, flow_label=0, l2_addr=00:c0:95:f8:4d:d3], > start_time=944207397.4, duration=0.0, service={ > }, history=Dd, uid=CcWDYV1vmfWIGfA3j7, tunnel=, > vlan=, inner_vlan=, dpd=, > conn=, extract_orig=F, extract_resp=F, > thresholds=, dce_rpc=, > dce_rpc_state=, dce_rpc_backing=, > dhcp=, dnp3=, dns=, > dns_state=, ftp=, ftp_data_reuse=F, > ssl=, http=, http_state=, > irc=, krb=, modbus=, > mysql=, ntlm=, radius=, > rdp=, rfb=, sip=, > sip_state=, snmp=, smtp=, > smtp_state=, socks=, ssh=, > syslog=] > 944207397.400000 internal error: bad reference count [2] > Aborted > > > Regards, > Binu > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170814/89cb0a2a/attachment.html From bill.de.ping at gmail.com Sun Aug 13 23:50:20 2017 From: bill.de.ping at gmail.com (william de ping) Date: Mon, 14 Aug 2017 09:50:20 +0300 Subject: [Bro] - delete a connection using connection_state_remove Message-ID: Hi everyone, In case I some test that occurs in http_request event. If the test returns FALSE, than I wish to delete this session\connection (I wish to save CPU time and prevent further processing of this packet or future session related packets). What would be the more appropriate way of accomplishing that ? Can I just call the connection_state_remove with my current c connection from my test function ? Or should I just "delete c;" ? Thanks B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170814/19826b55/attachment.html From jazoff at illinois.edu Mon Aug 14 06:45:03 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 14 Aug 2017 13:45:03 +0000 Subject: [Bro] Email Notice attempt #2 In-Reply-To: References: Message-ID: > On Aug 12, 2017, at 1:11 PM, craig bowser wrote: > > OK, further refinement: > > I've been going over the documentation for notices and raising alerts and googling and I think I've got the right code syntax, but I still am not getting emailed alerts. > > > I configured the following line in /usr/local/bro/share/bro/base/frameworks/notice/main.bro > > const mail_dest = "reswob10 at gmail.com" &redef; Do not modify the scripts that are shipped with bro. This setting is best configured by using the MailTo setting in the broctl.cfg > hook Notice::policy(n: Notice::Info) { > if (n$note == HTTP::Basic_Auth_Server) > print n$note; > add n$actions[Notice::ACTION_EMAIL]; > } You're missing a set of braces here and this block is really hook Notice::policy(n: Notice::Info) { if (n$note == HTTP::Basic_Auth_Server) print n$note; add n$actions[Notice::ACTION_EMAIL]; } You should use simply the redef Notice::emailed_types += { HTTP::Basic_Auth_Server }; that you had before. > When I run the script against a local pcap with HTTP basic auth traffic, I get the printed line: > > root at raspberrypi:/home/pi# bro -C -r http-bro.pcap http-auth-notice.bro > HTTP::Basic_Auth_Server > > > But I never get the emailed alert. Bro does not send email when running against a pcap file. > Could you please provide any tips/suggestions/corrections/rebukes for what I'm doing? Are there other configurations I missed? Is there a general log for bro that can show an error if it's trying to send email but it can't? Look at the notice.log. If the actions column contains Notice::ACTION_EMAIL then the script is trying to email the notice. If you did not receive the email then look at stderr.log and the system wide mail log. -- - Justin Azoff From jlay at slave-tothe-box.net Mon Aug 14 11:07:55 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Mon, 14 Aug 2017 12:07:55 -0600 Subject: [Bro] So before I upgrade (pf_ring revisit) Message-ID: <60743c78c8cdb4f431a4f59017db6d7f@localhost> As I recall last time around, there was a pf_ring vs. bro pf_ring plugin thread going on a while ago. I thought I'd revisit this since I'm going from 2.5.0 to 2.5.1. So..here's what I have: https://www.bro.org/documentation/load-balancing.html I believe the above is complete yes? The only question I would have answered here is what options to pass via command line. https://www.bro.org/sphinx/components/bro-plugins/pf_ring/README.html So THIS link is for using the plugin...this has info about starting command line, but not via node.cfg. I'm assuming I could use: pf_ring::eth0 instead of interface? Or do I just use what's in the load-balancing link? Additionally, what's the benefit of using one over the other method? Thank you. James -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170814/a1b6bb34/attachment.html From reswob10 at gmail.com Mon Aug 14 20:08:10 2017 From: reswob10 at gmail.com (craig bowser) Date: Mon, 14 Aug 2017 23:08:10 -0400 Subject: [Bro] Email Notice attempt #2 In-Reply-To: References: Message-ID: Thanks for the tips, thanks for your patience. I got the email to work. Craig L Bowser ____________________________ This email is measured by size. Bits and bytes may have settled during transport. On Mon, Aug 14, 2017 at 9:45 AM, Azoff, Justin S wrote: > > > On Aug 12, 2017, at 1:11 PM, craig bowser wrote: > > > > OK, further refinement: > > > > I've been going over the documentation for notices and raising alerts > and googling and I think I've got the right code syntax, but I still am not > getting emailed alerts. > > > > > > I configured the following line in /usr/local/bro/share/bro/base/ > frameworks/notice/main.bro > > > > const mail_dest = "reswob10 at gmail.com" > &redef; > > Do not modify the scripts that are shipped with bro. This setting is best > configured by using the MailTo setting in the broctl.cfg > > > hook Notice::policy(n: Notice::Info) { > > if (n$note == HTTP::Basic_Auth_Server) > > print n$note; > > add n$actions[Notice::ACTION_EMAIL]; > > } > > You're missing a set of braces here and this block is really > > hook Notice::policy(n: Notice::Info) { > if (n$note == HTTP::Basic_Auth_Server) > print n$note; > add n$actions[Notice::ACTION_EMAIL]; > } > > You should use simply the > > redef Notice::emailed_types += { HTTP::Basic_Auth_Server }; > > that you had before. > > > When I run the script against a local pcap with HTTP basic auth traffic, > I get the printed line: > > > > root at raspberrypi:/home/pi# bro -C -r http-bro.pcap http-auth-notice.bro > > HTTP::Basic_Auth_Server > > > > > > But I never get the emailed alert. > > Bro does not send email when running against a pcap file. > > > Could you please provide any tips/suggestions/corrections/rebukes for > what I'm doing? Are there other configurations I missed? Is there a > general log for bro that can show an error if it's trying to send email but > it can't? > > Look at the notice.log. If the actions column contains > Notice::ACTION_EMAIL then the script is trying to email the notice. If you > did not receive the email then look at stderr.log and the system wide mail > log. > > > > -- > - Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170814/71c59cc8/attachment.html From nahum at us.ibm.com Tue Aug 15 08:24:21 2017 From: nahum at us.ibm.com (Erich M Nahum) Date: Tue, 15 Aug 2017 11:24:21 -0400 Subject: [Bro] which kafka plugin to use? In-Reply-To: References: Message-ID: > I do plan to go further and use the Kafka plugin to pipe into Logstash, > but haven't gotten that far yet. I'm now trying to integrate with Kafka but have questions. The original kafka plugin, hosted at https://github.com/bro/bro-plugins, is now gone. This version built for me but I made the mistake of doing a git pull. All that's left is a message that one should now use the package manager. But there is no package for this yet. When trying to build from the git tree at https://github.com/g-clef/KafkaLogger, I get the following build error: [ 33%] Building CXX object CMakeFiles/Kafka-KafkaWriter.linux-x86_64.dir/src/AddingJson.cc.o /usr/src/KafkaLogger/src/AddingJson.cc:3:20: fatal error: config.h: No such file or directory compilation terminated. CMakeFiles/Kafka-KafkaWriter.linux-x86_64.dir/build.make:80: recipe for target 'CMakeFiles/Kafka-KafkaWriter.linux-x86_64.dir/src/AddingJson.cc.o' failed I don't see any config.h. I've tried building (configure --bro-dist) with both bro-2.5 and bro-2.5.1 I see there's now a Metro fork of the kafka plugin at https://github.com/apache/metron/tree/master/metron-sensors/bro-plugin-kafka but I am reluctant to try it based on email comments that it is beta. Any comments/suggestions? Thanks, -Erich -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170815/da07350d/attachment.html From nahum at us.ibm.com Tue Aug 15 08:48:53 2017 From: nahum at us.ibm.com (Erich M Nahum) Date: Tue, 15 Aug 2017 11:48:53 -0400 Subject: [Bro] which kafka plugin to use? In-Reply-To: References: Message-ID: > The original kafka plugin, hosted at https://github.com/bro/bro-plugins > , is now gone. D'oh, I now see it is also available in aux/plugins/kafka > When trying to build from the git tree at https://github.com/g-clef/ > KafkaLogger, > I get the following build error: > > [ 33%] Building CXX object CMakeFiles/Kafka-KafkaWriter.linux- > x86_64.dir/src/AddingJson.cc.o > /usr/src/KafkaLogger/src/AddingJson.cc:3:20: fatal error: config.h: > No such file or directory > compilation terminated. > CMakeFiles/Kafka-KafkaWriter.linux-x86_64.dir/build.make:80: recipe > for target 'CMakeFiles/Kafka-KafkaWriter.linux-x86_64.dir/src/ > AddingJson.cc.o' failed Perhaps this is useful to Aaron Gee-Clough. I forgot to mention that I'm using Ubuntu 16.04 running apt-get upgrade periodically. > I see there's now a Metro fork of the kafka plugin at > > https://github.com/apache/metron/tree/master/metron-sensors/bro-plugin-kafka > > but I am reluctant to try it based on email comments that it is beta. > > Any comments/suggestions? While I can use the version in the bro source, I guess my question still stands: what's the long-term outlook for kafka support? -Erich -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170815/d2316812/attachment.html From waa at cs.umd.edu Tue Aug 15 09:08:20 2017 From: waa at cs.umd.edu (William Arbaugh) Date: Tue, 15 Aug 2017 18:08:20 +0200 Subject: [Bro] which kafka plugin to use? In-Reply-To: References: Message-ID: > > Any comments/suggestions? While I can use the version in the bro source, I guess my question still stands: what's the long-term outlook for kafka support? For what it?s worth, we use filebeat to shoot our bro logs into Kafka. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170815/ad801163/attachment.html From zeolla at gmail.com Tue Aug 15 09:25:28 2017 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Tue, 15 Aug 2017 16:25:28 +0000 Subject: [Bro] which kafka plugin to use? In-Reply-To: References: Message-ID: To clarify, the Metron project developed the kafka plugin for its own uses and then contributed it into bro-plugins. Recently I worked with the initial creator of the plugin to unify all of the updates that have happened to it over the years (in a way that complies with its LICENSE) here . I'm in the process of porting it to be a bro package and moving it to https://github.com/apache/metron-bro-plugin-kafka which will be its final resting point. I'm currently battling through some CentOS 6 -> 7 upgrades in Metron, and then upgrading bro to 2.5.1 (from 2.4) in Metron (and all of the associated automation/testing), and then finally I will be publishing the kafka plugin module and submitting a PR to https://github.com/bro/packages. Some very, *very* early movement towards packaging the kafka plugin can be found here (caution, it almost definitely does not work - I'm trying to figure out how to handle the librdkafka dependancy in the package, any feedback would be helpful). I would /love/ to have this ready to go for brocon (which is my goal). Jon On Tue, Aug 15, 2017 at 12:00 PM Erich M Nahum wrote: > > The original kafka plugin, hosted at https://github.com/bro/bro-plugins > > , is now gone. > > D'oh, I now see it is also available in aux/plugins/kafka > > > > > When trying to build from the git tree at https://github.com/g-clef/ > > KafkaLogger, > > I get the following build error: > > > > [ 33%] Building CXX object CMakeFiles/Kafka-KafkaWriter.linux- > > x86_64.dir/src/AddingJson.cc.o > > /usr/src/KafkaLogger/src/AddingJson.cc:3:20: fatal error: config.h: > > No such file or directory > > compilation terminated. > > CMakeFiles/Kafka-KafkaWriter.linux-x86_64.dir/build.make:80: recipe > > for target 'CMakeFiles/Kafka-KafkaWriter.linux-x86_64.dir/src/ > > AddingJson.cc.o' failed > > Perhaps this is useful to Aaron Gee-Clough. I forgot to mention that > I'm using Ubuntu 16.04 running apt-get upgrade periodically. > > > > > I see there's now a Metro fork of the kafka plugin at > > > > > https://github.com/apache/metron/tree/master/metron-sensors/bro-plugin-kafka > > > > but I am reluctant to try it based on email comments that it is beta. > > > > Any comments/suggestions? > > While I can use the version in the bro source, I guess my question still > stands: > what's the long-term outlook for kafka support? > > > > -Erich > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170815/ae8283d3/attachment-0001.html From zeolla at gmail.com Tue Aug 15 09:52:29 2017 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Tue, 15 Aug 2017 16:52:29 +0000 Subject: [Bro] which kafka plugin to use? In-Reply-To: References: Message-ID: For what it's worth, I'm currently using the plugin available under https://github.com/apache/metron/tree/master/metron-sensors/bro-plugin-kafka in my production bro environment, which is an 8 node cluster with > 25,000 events per second and it's working just fine for me, but I would love to get others to test it. I'm not making any changes to the core kafka plugin itself for the move, just packaging it and incrementing some version numbers - the real heavy lift is within Metron itself, not the bro plugin. Jon On Tue, Aug 15, 2017 at 12:25 PM Zeolla at GMail.com wrote: > To clarify, the Metron project developed the kafka plugin for its own uses > and then contributed it into bro-plugins. Recently I worked with the > initial creator of the plugin to unify all of the updates that have > happened to it over the years (in a way that complies with its LICENSE) > here > > . > > I'm in the process of porting it to be a bro package and moving it to > https://github.com/apache/metron-bro-plugin-kafka which will be its final > resting point. I'm currently battling through some CentOS 6 -> 7 upgrades > in Metron, and then upgrading bro to 2.5.1 (from 2.4) in Metron (and all of > the associated automation/testing), and then finally I will be publishing > the kafka plugin module and submitting a PR to > https://github.com/bro/packages. Some very, *very* early movement > towards packaging the kafka plugin can be found here > (caution, it > almost definitely does not work - I'm trying to figure out how to handle > the librdkafka dependancy in the package, any feedback would be helpful). > > I would /love/ to have this ready to go for brocon (which is my goal). > > Jon > > On Tue, Aug 15, 2017 at 12:00 PM Erich M Nahum wrote: > >> > The original kafka plugin, hosted at https://github.com/bro/bro-plugins >> > , is now gone. >> >> D'oh, I now see it is also available in aux/plugins/kafka >> >> >> >> > When trying to build from the git tree at https://github.com/g-clef/ >> > KafkaLogger, >> > I get the following build error: >> > >> > [ 33%] Building CXX object CMakeFiles/Kafka-KafkaWriter.linux- >> > x86_64.dir/src/AddingJson.cc.o >> > /usr/src/KafkaLogger/src/AddingJson.cc:3:20: fatal error: config.h: >> > No such file or directory >> > compilation terminated. >> > CMakeFiles/Kafka-KafkaWriter.linux-x86_64.dir/build.make:80: recipe >> > for target 'CMakeFiles/Kafka-KafkaWriter.linux-x86_64.dir/src/ >> > AddingJson.cc.o' failed >> >> Perhaps this is useful to Aaron Gee-Clough. I forgot to mention that >> I'm using Ubuntu 16.04 running apt-get upgrade periodically. >> >> >> >> > I see there's now a Metro fork of the kafka plugin at >> > >> > >> https://github.com/apache/metron/tree/master/metron-sensors/bro-plugin-kafka >> > >> > but I am reluctant to try it based on email comments that it is beta. >> > >> > Any comments/suggestions? >> >> While I can use the version in the bro source, I guess my question still >> stands: >> what's the long-term outlook for kafka support? >> >> >> >> -Erich >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -- > > Jon > -- Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170815/b1dfbef5/attachment.html From bill.de.ping at gmail.com Wed Aug 16 05:32:16 2017 From: bill.de.ping at gmail.com (william de ping) Date: Wed, 16 Aug 2017 15:32:16 +0300 Subject: [Bro] - ICMP not in conn.log Message-ID: Hi all, Anyone knows why ICMP does not appear in conn.log ? I see that the plugin and icmp analyzer are loaded, yet no indication in conn.log for proto icmp. Thanks B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170816/09b0751c/attachment.html From reswob10 at gmail.com Wed Aug 16 12:36:50 2017 From: reswob10 at gmail.com (craig bowser) Date: Wed, 16 Aug 2017 15:36:50 -0400 Subject: [Bro] capture loss vs dropped packets Message-ID: According to the following: https://www.bro.org/documentation/faq.html#how-can-i-reduce-the-amount-of-captureloss-or-dropped-packets-notices http://blog.securityonion.net/2011/10/when-is-full-packet-capture-not-full.html I can get capture loss notices when an bro isn't getting all the acks from an upstream device (network tap, wrongly configured ethernet port, etc) which is different from dropped packets which is when bro can't process all the packets it sees. So in my environment, I'm getting entries in the capture-loss.log, but I'm not getting any corresponding entries in my notice.log. https://www.bro.org/sphinx/scripts/policy/misc/capture-loss.bro.html Does this mean that I'm seeing Capture Loss without Dropped Packets? and that it's caused by a device upstream to Bro? Thanks. Craig L Bowser ____________________________ This email is measured by size. Bits and bytes may have settled during transport. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170816/730a17f6/attachment.html From philosnef at gmail.com Thu Aug 17 04:14:50 2017 From: philosnef at gmail.com (erik clark) Date: Thu, 17 Aug 2017 07:14:50 -0400 Subject: [Bro] looping traffic and bpf Message-ID: I foresee a problem in the very near future where I am sending traffic out to our splunk indexers over the same network I am tapping. I am pretty sure this would loop the traffic through the tap, and don't want to do that. I see a wide variety of ways to run bpf statements from 5 years ago till somewhat recently in google. What is the best way in 2.5 to strip a single address from bros inspection with a filter? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170817/43569971/attachment.html From brandonlattin at gmail.com Thu Aug 17 07:13:37 2017 From: brandonlattin at gmail.com (Brandon Lattin) Date: Thu, 17 Aug 2017 09:13:37 -0500 Subject: [Bro] looping traffic and bpf In-Reply-To: References: Message-ID: not host You can get significantly more fancy as necessary: https://biot.com/capstats/bpf.html https://www.bro.org/sphinx/scripts/base/frameworks/packet-filter/main.bro.html On Thu, Aug 17, 2017 at 6:14 AM, erik clark wrote: > I foresee a problem in the very near future where I am sending traffic out > to our splunk indexers over the same network I am tapping. I am pretty sure > this would loop the traffic through the tap, and don't want to do that. > > I see a wide variety of ways to run bpf statements from 5 years ago till > somewhat recently in google. What is the best way in 2.5 to strip a single > address from bros inspection with a filter? > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170817/b7368e04/attachment.html From seth at corelight.com Thu Aug 17 08:57:34 2017 From: seth at corelight.com (Seth Hall) Date: Thu, 17 Aug 2017 11:57:34 -0400 Subject: [Bro] - ICMP not in conn.log In-Reply-To: References: Message-ID: On 16 Aug 2017, at 8:32, william de ping wrote: > Anyone knows why ICMP does not appear in conn.log ? > I see that the plugin and icmp analyzer are loaded, yet no indication > in > conn.log for proto icmp. We never created a model to include that data. I've had it on my radar for a while, but unfortunately have not found the time to add it. I was hoping to figure out a way to include the data in conn.log for connections where ICMP was seen in relation to a connection (like Time Exceeded) instead of just having them listed as separate "connections". .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From seth at corelight.com Thu Aug 17 09:13:09 2017 From: seth at corelight.com (Seth Hall) Date: Thu, 17 Aug 2017 12:13:09 -0400 Subject: [Bro] So before I upgrade (pf_ring revisit) In-Reply-To: <60743c78c8cdb4f431a4f59017db6d7f@localhost> References: <60743c78c8cdb4f431a4f59017db6d7f@localhost> Message-ID: Sorry for the delay in answering! On 14 Aug 2017, at 14:07, James Lay wrote: > pf_ring::eth0 > > instead of interface? Or do I just use what's in the load-balancing > link? Additionally, what's the benefit of using one over the other > method? Thank you. I think you may want to use whats in the load-balancing link (the libpcap wrapper approach). I'm not sure of the state of the pf_ring plugin, especially since we removed everything in the bro-plugins repository. Robin emailed package maintainers there to try and get them to create Bro packages and I'm not sure the pf_ring one was moved over. The other problem is that the pf_ring plugin didn't/doesn't have a broctl plugin so you can't change any settings (such as the app_id). I *think* that load balancing works with it though, but I'm not sure offhand how you'd configure it correctly with broctl. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From jlay at slave-tothe-box.net Thu Aug 17 10:46:16 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 17 Aug 2017 11:46:16 -0600 Subject: [Bro] So before I upgrade (pf_ring revisit) In-Reply-To: References: <60743c78c8cdb4f431a4f59017db6d7f@localhost> Message-ID: <21138e3c2b2aeea7b2e02d2b3dfbbb17@localhost> On 2017-08-17 10:13, Seth Hall wrote: > Sorry for the delay in answering! > > On 14 Aug 2017, at 14:07, James Lay wrote: > >> pf_ring::eth0 >> >> instead of interface? Or do I just use what's in the load-balancing >> link? Additionally, what's the benefit of using one over the other >> method? Thank you. > > I think you may want to use whats in the load-balancing link (the > libpcap wrapper approach). > > I'm not sure of the state of the pf_ring plugin, especially since we > removed everything in the bro-plugins repository. Robin emailed > package maintainers there to try and get them to create Bro packages > and I'm not sure the pf_ring one was moved over. The other problem is > that the pf_ring plugin didn't/doesn't have a broctl plugin so you > can't change any settings (such as the app_id). I *think* that load > balancing works with it though, but I'm not sure offhand how you'd > configure it correctly with broctl. > > .Seth > > -- > Seth Hall * Corelight, Inc * www.corelight.com Awesome...thanks Seth I'll stick with the load-balancing link for now. James From alkenepan at gmail.com Fri Aug 18 00:21:53 2017 From: alkenepan at gmail.com (Alkene Pan) Date: Fri, 18 Aug 2017 15:21:53 +0800 Subject: [Bro] TCP reassembly question [Port Traffic Mirroring] Message-ID: I encountered a tcp reassembly problem, that begets the bro parse the tcp stream fail. Sincere search for solutions. The following is a description of the problem. There are two pcap file, the pcap data capture from switch(Port Traffic Mirroring). As below: - pcap1, we are use bro to parse the pcap1 file. Been testing, bro can not log http request. I am not sure whether the TCP stream has some messy. - pcap2, base on the pcap1, we were use wireshark to deleted a packet and generated another new pcap2 file. The deleted packet's status was described as "[TCP ACKed unseen segment]" in wireshark. The bro parse the pcap correctly. Is there any suggestion to solve the problem. Thanks very much. Below listed the test step and results: Bro with capture-loss loaded: root at sensor ~/temp# cat loss.bro @load misc/capture-loss.bro ==================================================================== PCAP1? root at sensor ~/temp# bro -C -r single_issue_00.pcap loss.bro root at sensor ~/temp# cat conn.log #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path conn #open 2017-08-18-13-23-54 #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig local_resp missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents #types time string addr port addr port enum string interval count count string bool bool count string count count count count set[string] 1502979585.855778 CPkvTTF91m8B1Yfdb 10.0.81.16 56144 10.0.81.48 80 tcp - 0.002211 90 11977 SF - - 90 ShAaDdfF 12 722 12 12609 (empty) #close 2017-08-18-13-23-54 root at sensor ~/temp# cat http.log #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path http #open 2017-08-18-13-23-54 #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p trans_depth method host uri referrer version user_agent request_body_len response_body_len status_code status_msg info_code info_msg tags username password proxied orig_fuids orig_filenames orig_mime_types resp_fuids resp_filenames resp_mime_types #types time string addr port addr port count string string string string string string count count count string count string set[enum] string string set[string] vector[string] vector[string] vector[string] vector[string] vector[string] vector[string] 1502979585.856854 CPkvTTF91m8B1Yfdb 10.0.81.16 56144 10.0.81.48 80 1 - - - - 1.1 - 0 11741 200OK - - (empty) - - - - - - FF9ypH1xMjwIUswLU1 - text/html #close 2017-08-18-13-23-54 root at sensor ~/temp# cat capture_loss.log #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path capture_loss #open 2017-08-18-13-23-54 #fields ts ts_delta peer gaps acks percent_lost #types time interval string count count double 1502979585.858078 0.002300 bro 1 6 16.666667 #close 2017-08-18-13-23-54 ==================================================================== after delete [TCP ACKed unseen segment] then PCAP2: root at sensor ~/temp# cat conn.log #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path conn #open 2017-08-18-13-54-49 #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig local_resp missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents #types time string addr port addr port enum string interval count count string bool bool count string count count count count set[string] 1502979585.855778 CbMbYjzyDnDallire 10.0.81.16 56144 10.0.81.48 80 tcp http 0.002211 90 11977 SF - - 0 ShADdfFa 12 722 11 12557 (empty) #close 2017-08-18-13-54-49 #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path http #open 2017-08-18-13-54-49 #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p trans_depth method host uri referrer version user_agent request_body_len response_body_len status_code status_msg info_code info_msg tags username password proxied orig_fuids orig_filenames orig_mime_types resp_fuids resp_filenames resp_mime_types #types time string addr port addr port count string string string string string string count count count string count string set[enum] string string set[string] vector[string] vector[string] vector[string] vector[string] vector[string] vector[string] 1502979585.856615 CbMbYjzyDnDallire 10.0.81.16 56144 10.0.81.48 80 1 GET 10.0.81.48 /pvs/233.html - 1.1 ApacheBench/2.3 0 11741 200 OK - - (empty) - - - -- - FF9ypH1xMjwIUswLU1 - text/html #close 2017-08-18-13-54-49 root at sensor ~/temp# cat capture_loss.log #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path capture_loss #open 2017-08-18-13-54-49 #fields ts ts_delta peer gaps acks percent_lost #types time interval string count count double 1502979585.858078 0.002300 bro 0 6 0.0 #close 2017-08-18-13-54-49 ==================================================================== Best regards, AlkenePan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170818/533f8b77/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: PACP1_single_issue_00.pcap Type: application/octet-stream Size: 14075 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170818/533f8b77/attachment-0002.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: PCAP2_single_issue_00_removed_bad_packet.pcap Type: application/octet-stream Size: 13993 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170818/533f8b77/attachment-0003.obj From pssunu6 at gmail.com Mon Aug 21 05:31:12 2017 From: pssunu6 at gmail.com (ps sunu) Date: Mon, 21 Aug 2017 18:01:12 +0530 Subject: [Bro] NFS analyser Message-ID: Hi Is NFS analyser is working in bro 2.5 ?? Regards, Sunu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170821/1c081f28/attachment.html From xuchen890530 at gmail.com Mon Aug 21 13:57:27 2017 From: xuchen890530 at gmail.com (Chen Xu) Date: Mon, 21 Aug 2017 16:57:27 -0400 Subject: [Bro] Run Python script in Bro Message-ID: Hello all, I use the str_shell_escape() to run a Python script using command line. I am wondering how I can get the return or print value of Python script, because I will need the result in a if statement in Bro. Thanks, Chen Xu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170821/cf14f97a/attachment.html From jan.grashoefer at gmail.com Tue Aug 22 00:44:25 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Tue, 22 Aug 2017 09:44:25 +0200 Subject: [Bro] Run Python script in Bro In-Reply-To: References: Message-ID: <02651e71-69a8-c1cb-d723-9a597e9d0893@gmail.com> Hi Chen, > I use the str_shell_escape() to run a Python script using command line. I > am wondering how I can get the return or print value of Python script, > because I will need the result in a if statement in Bro. str_shell_escape() is used to escape strings in that context. I think system() is what you are looking for: https://www.bro.org/sphinx/scripts/base/bif/bro.bif.bro.html#id-system Jan From jdopheid at illinois.edu Tue Aug 22 08:53:20 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Tue, 22 Aug 2017 15:53:20 +0000 Subject: [Bro] BroCon lightning talk sign-up Message-ID: <2DFDC0E8-F16C-4A8E-BFF2-74136CA37541@illinois.edu> Pinging this thread again. We?re aiming for nine talks, we have six signed up so far. If you have something you would like to share, please sign up. ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign On 8/1/17, 12:07 PM, "bro-bounces at bro.org on behalf of Dopheide, Jeannette M" wrote: Hello Bro Community, In order to queue up requests to speak at the BroCon Lightning Talk session, we?ve created a Google form: https://goo.gl/forms/VetHBj0lQ2P2DZik2 We ask for your name, email address, organization, talk title, contact info, and a link to 1 or 2 slides to help us generate the slide deck. Lightning talk abstract: Have you contributed a package to the new Bro Package Manager? Or do you have something interesting to share related to Bro that doesn?t fit into a traditional presentation? Great! We?d love to hear from you. We have scheduled a 45-minute session for lightning talks. Be prepared to quickly identify your point, demonstrate it, and provide a link or contact info for later follow-up. We?d like to accommodate as many talks as possible so please limit your talk to less than 5 minutes. No commercially incentivized presentations, please. Register for BroCon: https://brocon2017.eventbrite.com/ Don?t forget to book your hotel: https://www.bro.org/community/brocon2017.html#hotelinformation The block of rooms expires on August 14th. See you in September, The Bro Project ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From seth at corelight.com Tue Aug 22 10:55:32 2017 From: seth at corelight.com (Seth Hall) Date: Tue, 22 Aug 2017 13:55:32 -0400 Subject: [Bro] Run Python script in Bro In-Reply-To: References: Message-ID: On 21 Aug 2017, at 16:57, Chen Xu wrote: > I use the str_shell_escape() to run a Python script using command > line. I > am wondering how I can get the return or print value of Python script, > because I will need the result in a if statement in Bro. The "Exec" library is what you want to use. You can use it like this... when ( local ret = Exec::run([$cmd="ls"]) ) { print ret$exit_code; print ret$stdout; } The only thing to keep in mind is that the body of "when" statements are only executed once the condition has completed. The body of the when statement will be "registered" and execution will immediately continue below the when statement so you many have to do something like have your if statement you want to do *inside* the when statement's body. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From vikrambasu059 at gmail.com Wed Aug 23 02:14:43 2017 From: vikrambasu059 at gmail.com (Vikram Basu) Date: Wed, 23 Aug 2017 14:44:43 +0530 Subject: [Bro] Combining fields from http.log and files.log Message-ID: <599d4784.c223620a.7d9a.5d9a@mx.google.com> Hi, Is it possible to create a combined log file which will have some fields from Bro?s http.log file and some from Bro?s file.log file using the `fuid` field as the common field ? When I download a file using HTTP connection, I get the name of the file in the http.log while the file size is present in the files.log How can I correlate the information into a single log file ? Regards Vikram Basu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170823/d8262c99/attachment.html From jlay at slave-tothe-box.net Thu Aug 24 08:56:04 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 24 Aug 2017 09:56:04 -0600 Subject: [Bro] Startup cleanup Message-ID: <1183ed12c0cb225a3d068d3662b65338@localhost> So here's my startup line and standard output sudo /usr/local/bro/bin/bro -C -i eth0 -i eth1 --filter 'not ((host x.x.x.x and net 192.168.1.0/24) and (tcp port or tcp port )) and not ip6' local "Site::local_nets += { x.x.x.x/32,192.168.1.0/24 }" internal warning in /usr/local/bro/share/bro/policy/frameworks/intel/seen/__load__.bro, line 1: Discarded extraneous Broxygen comment: check link in mail_links internal warning in /usr/local/bro/share/bro/policy/frameworks/intel/seen/__load__.bro, line 1: Discarded extraneous Broxygen comment: for internal warning in /usr/local/bro/share/bro/policy/frameworks/intel/seen/__load__.bro, line 1: Discarded extraneous Broxygen comment: print fmt ("log_mine Log_mime: %s", rec); internal warning in /usr/local/bro/share/bro/policy/frameworks/intel/seen/__load__.bro, line 1: Discarded extraneous Broxygen comment: aashish: need to port to file analysis framework internal warning in /usr/local/bro/share/bro/policy/frameworks/intel/seen/./conn-established.bro, line 1: Discarded extraneous Broxygen comment: check link in mail_links internal warning in /usr/local/bro/share/bro/policy/frameworks/intel/seen/./conn-established.bro, line 1: Discarded extraneous Broxygen comment: for internal warning in /usr/local/bro/share/bro/policy/frameworks/intel/seen/./conn-established.bro, line 1: Discarded extraneous Broxygen comment: print fmt ("log_mine Log_mime: %s", rec); internal warning in /usr/local/bro/share/bro/policy/frameworks/intel/seen/./conn-established.bro, line 1: Discarded extraneous Broxygen comment: aashish: need to port to file analysis framework internal warning in /usr/local/bro/share/bro/policy/frameworks/intel/seen/./conn-established.bro, line 1: Discarded extraneous Broxygen comment: check link in mail_links internal warning in /usr/local/bro/share/bro/policy/frameworks/intel/seen/./conn-established.bro, line 1: Discarded extraneous Broxygen comment: for internal warning in /usr/local/bro/share/bro/policy/frameworks/intel/seen/./conn-established.bro, line 1: Discarded extraneous Broxygen comment: print fmt ("log_mine Log_mime: %s", rec); internal warning in /usr/local/bro/share/bro/policy/frameworks/intel/seen/./conn-established.bro, line 1: Discarded extraneous Broxygen comment: aashish: need to port to file analysis framework internal warning in /usr/local/bro/share/bro/policy/frameworks/intel/seen/././where-locations.bro, line 1: Discarded extraneous Broxygen comment: check link in mail_links internal warning in /usr/local/bro/share/bro/policy/frameworks/intel/seen/././where-locations.bro, line 1: Discarded extraneous Broxygen comment: for internal warning in /usr/local/bro/share/bro/policy/frameworks/intel/seen/././where-locations.bro, line 1: Discarded extraneous Broxygen comment: print fmt ("log_mine Log_mime: %s", rec); internal warning in /usr/local/bro/share/bro/policy/frameworks/intel/seen/././where-locations.bro, line 1: Discarded extraneous Broxygen comment: aashish: need to port to file analysis framework internal warning in /usr/local/bro/share/bro/policy/frameworks/intel/seen/././where-locations.bro, line 1: Discarded extraneous Broxygen comment: check link in mail_links internal warning in /usr/local/bro/share/bro/policy/frameworks/intel/seen/././where-locations.bro, line 1: Discarded extraneous Broxygen comment: for internal warning in /usr/local/bro/share/bro/policy/frameworks/intel/seen/././where-locations.bro, line 1: Discarded extraneous Broxygen comment: print fmt ("log_mine Log_mime: %s", rec); internal warning in /usr/local/bro/share/bro/policy/frameworks/intel/seen/././where-locations.bro, line 1: Discarded extraneous Broxygen comment: aashish: need to port to file analysis framework , line 1: listening on eth0 , line 1: listening on eth1 1503589314.254774 error in , line 1: Bad IP address: 5 1503589314.254774 error in , line 1: Bad IP address: 6 1503589314.254774 error in , line 1: Bad IP address: 1 Anything I need to be concerned about here? Thank you. James From dopheide at gmail.com Thu Aug 24 09:18:56 2017 From: dopheide at gmail.com (Mike Dopheide) Date: Thu, 24 Aug 2017 11:18:56 -0500 Subject: [Bro] Startup cleanup In-Reply-To: <1183ed12c0cb225a3d068d3662b65338@localhost> References: <1183ed12c0cb225a3d068d3662b65338@localhost> Message-ID: Not sure about that end bit, but you can ignore all the "extraneous Broxygen comment" lines. That basically just means someone (Aashish!) used two ##'s to start a comment. It's a habit I have as well so I see those all the time. -Dop On Thu, Aug 24, 2017 at 10:56 AM, James Lay wrote: > So here's my startup line and standard output > > sudo /usr/local/bro/bin/bro -C -i eth0 -i eth1 --filter 'not ((host > x.x.x.x and net 192.168.1.0/24) and (tcp port or tcp port > )) and not ip6' local "Site::local_nets += { x.x.x.x/32,192.168.1.0/24 > }" > internal warning in > /usr/local/bro/share/bro/policy/frameworks/intel/seen/__load__.bro, line > 1: Discarded extraneous Broxygen comment: check link in mail_links > internal warning in > /usr/local/bro/share/bro/policy/frameworks/intel/seen/__load__.bro, line > 1: Discarded extraneous Broxygen comment: for > internal warning in > /usr/local/bro/share/bro/policy/frameworks/intel/seen/__load__.bro, line > 1: Discarded extraneous Broxygen comment: print fmt ("log_mine > Log_mime: %s", rec); > internal warning in > /usr/local/bro/share/bro/policy/frameworks/intel/seen/__load__.bro, line > 1: Discarded extraneous Broxygen comment: aashish: need to port to file > analysis framework > internal warning in > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > /conn-established.bro, > line 1: Discarded extraneous Broxygen comment: check link in mail_links > internal warning in > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > /conn-established.bro, > line 1: Discarded extraneous Broxygen comment: for > internal warning in > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > /conn-established.bro, > line 1: Discarded extraneous Broxygen comment: print fmt > ("log_mine Log_mime: %s", rec); > internal warning in > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > /conn-established.bro, > line 1: Discarded extraneous Broxygen comment: aashish: need to port to > file analysis framework > internal warning in > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > /conn-established.bro, > line 1: Discarded extraneous Broxygen comment: check link in mail_links > internal warning in > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > /conn-established.bro, > line 1: Discarded extraneous Broxygen comment: for > internal warning in > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > /conn-established.bro, > line 1: Discarded extraneous Broxygen comment: print fmt > ("log_mine Log_mime: %s", rec); > internal warning in > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > /conn-established.bro, > line 1: Discarded extraneous Broxygen comment: aashish: need to port to > file analysis framework > internal warning in > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > /./where-locations.bro, > line 1: Discarded extraneous Broxygen comment: check link in mail_links > internal warning in > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > /./where-locations.bro, > line 1: Discarded extraneous Broxygen comment: for > internal warning in > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > /./where-locations.bro, > line 1: Discarded extraneous Broxygen comment: print fmt > ("log_mine Log_mime: %s", rec); > internal warning in > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > /./where-locations.bro, > line 1: Discarded extraneous Broxygen comment: aashish: need to port to > file analysis framework > internal warning in > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > /./where-locations.bro, > line 1: Discarded extraneous Broxygen comment: check link in mail_links > internal warning in > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > /./where-locations.bro, > line 1: Discarded extraneous Broxygen comment: for > internal warning in > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > /./where-locations.bro, > line 1: Discarded extraneous Broxygen comment: print fmt > ("log_mine Log_mime: %s", rec); > internal warning in > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > /./where-locations.bro, > line 1: Discarded extraneous Broxygen comment: aashish: need to port to > file analysis framework > , line 1: listening on eth0 > > , line 1: listening on eth1 > > 1503589314.254774 error in , line 1: Bad IP address: 5 > 1503589314.254774 error in , line 1: Bad IP address: 6 > 1503589314.254774 error in , line 1: Bad IP address: 1 > > Anything I need to be concerned about here? Thank you. > > James > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170824/d3a3fcdb/attachment.html From asharma at lbl.gov Thu Aug 24 10:15:16 2017 From: asharma at lbl.gov (Aashish Sharma) Date: Thu, 24 Aug 2017 10:15:16 -0700 Subject: [Bro] Startup cleanup In-Reply-To: References: <1183ed12c0cb225a3d068d3662b65338@localhost> Message-ID: <20170824171514.GH58687@MacPro.local> > Broxygen comment" lines. That basically just means someone (Aashish!) used ( I knew had to be me involved somewhere - apologies ) I think Broxygen pointing to intel-framework is merely an artifact. Issue is you are running smtp-embedded-urls-bloom.bro where I have "##" in comments and broxygen doesn't like it. (this script is rather quite old version) I believe have cleaned up code here: https://github.com/initconf/smtp-analysis I'll send a followup email in a little bit with a link to more latest stuff. Just want ot make sure its cleaned up before I share the link with you. Aashish On Thu, Aug 24, 2017 at 11:18:56AM -0500, Mike Dopheide wrote: > Not sure about that end bit, but you can ignore all the "extraneous > Broxygen comment" lines. That basically just means someone (Aashish!) used > two ##'s to start a comment. It's a habit I have as well so I see those > all the time. > > -Dop > > On Thu, Aug 24, 2017 at 10:56 AM, James Lay > wrote: > > > So here's my startup line and standard output > > > > sudo /usr/local/bro/bin/bro -C -i eth0 -i eth1 --filter 'not ((host > > x.x.x.x and net 192.168.1.0/24) and (tcp port or tcp port > > )) and not ip6' local "Site::local_nets += { x.x.x.x/32,192.168.1.0/24 > > }" > > internal warning in > > /usr/local/bro/share/bro/policy/frameworks/intel/seen/__load__.bro, line > > 1: Discarded extraneous Broxygen comment: check link in mail_links > > internal warning in > > /usr/local/bro/share/bro/policy/frameworks/intel/seen/__load__.bro, line > > 1: Discarded extraneous Broxygen comment: for > > internal warning in > > /usr/local/bro/share/bro/policy/frameworks/intel/seen/__load__.bro, line > > 1: Discarded extraneous Broxygen comment: print fmt ("log_mine > > Log_mime: %s", rec); > > internal warning in > > /usr/local/bro/share/bro/policy/frameworks/intel/seen/__load__.bro, line > > 1: Discarded extraneous Broxygen comment: aashish: need to port to file > > analysis framework > > internal warning in > > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > > /conn-established.bro, > > line 1: Discarded extraneous Broxygen comment: check link in mail_links > > internal warning in > > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > > /conn-established.bro, > > line 1: Discarded extraneous Broxygen comment: for > > internal warning in > > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > > /conn-established.bro, > > line 1: Discarded extraneous Broxygen comment: print fmt > > ("log_mine Log_mime: %s", rec); > > internal warning in > > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > > /conn-established.bro, > > line 1: Discarded extraneous Broxygen comment: aashish: need to port to > > file analysis framework > > internal warning in > > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > > /conn-established.bro, > > line 1: Discarded extraneous Broxygen comment: check link in mail_links > > internal warning in > > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > > /conn-established.bro, > > line 1: Discarded extraneous Broxygen comment: for > > internal warning in > > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > > /conn-established.bro, > > line 1: Discarded extraneous Broxygen comment: print fmt > > ("log_mine Log_mime: %s", rec); > > internal warning in > > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > > /conn-established.bro, > > line 1: Discarded extraneous Broxygen comment: aashish: need to port to > > file analysis framework > > internal warning in > > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > > /./where-locations.bro, > > line 1: Discarded extraneous Broxygen comment: check link in mail_links > > internal warning in > > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > > /./where-locations.bro, > > line 1: Discarded extraneous Broxygen comment: for > > internal warning in > > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > > /./where-locations.bro, > > line 1: Discarded extraneous Broxygen comment: print fmt > > ("log_mine Log_mime: %s", rec); > > internal warning in > > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > > /./where-locations.bro, > > line 1: Discarded extraneous Broxygen comment: aashish: need to port to > > file analysis framework > > internal warning in > > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > > /./where-locations.bro, > > line 1: Discarded extraneous Broxygen comment: check link in mail_links > > internal warning in > > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > > /./where-locations.bro, > > line 1: Discarded extraneous Broxygen comment: for > > internal warning in > > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > > /./where-locations.bro, > > line 1: Discarded extraneous Broxygen comment: print fmt > > ("log_mine Log_mime: %s", rec); > > internal warning in > > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. > > /./where-locations.bro, > > line 1: Discarded extraneous Broxygen comment: aashish: need to port to > > file analysis framework > > , line 1: listening on eth0 > > > > , line 1: listening on eth1 > > > > 1503589314.254774 error in , line 1: Bad IP address: 5 > > 1503589314.254774 error in , line 1: Bad IP address: 6 > > 1503589314.254774 error in , line 1: Bad IP address: 1 > > > > Anything I need to be concerned about here? Thank you. > > > > James > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jlay at slave-tothe-box.net Thu Aug 24 13:15:10 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 24 Aug 2017 14:15:10 -0600 Subject: [Bro] Startup cleanup In-Reply-To: <20170824171514.GH58687@MacPro.local> References: <1183ed12c0cb225a3d068d3662b65338@localhost> <20170824171514.GH58687@MacPro.local> Message-ID: <611b7e627daded4df17e454e7c9c4bf8@localhost> On 2017-08-24 11:15, Aashish Sharma wrote: >> Broxygen comment" lines. That basically just means someone (Aashish!) >> used > ( I knew had to be me involved somewhere - apologies ) > > I think Broxygen pointing to intel-framework is merely an artifact. > Issue is you > are running smtp-embedded-urls-bloom.bro where I have "##" in comments > and > broxygen doesn't like it. (this script is rather quite old version) > > I believe have cleaned up code here: > > https://github.com/initconf/smtp-analysis > > I'll send a followup email in a little bit with a link to more latest > stuff. > Just want ot make sure its cleaned up before I share the link with you. > > Aashish Good on ya thanks Aashish as well as others that have responded. Figured I'd make sure I had less to look through as I go through the upgrade process. James > > > On Thu, Aug 24, 2017 at 11:18:56AM -0500, Mike Dopheide wrote: >> Not sure about that end bit, but you can ignore all the "extraneous >> Broxygen comment" lines. That basically just means someone (Aashish!) >> used >> two ##'s to start a comment. It's a habit I have as well so I see >> those >> all the time. >> >> -Dop >> >> On Thu, Aug 24, 2017 at 10:56 AM, James Lay >> wrote: >> >> > So here's my startup line and standard output >> > >> > sudo /usr/local/bro/bin/bro -C -i eth0 -i eth1 --filter 'not ((host >> > x.x.x.x and net 192.168.1.0/24) and (tcp port or tcp port >> > )) and not ip6' local "Site::local_nets += { x.x.x.x/32,192.168.1.0/24 >> > }" >> > internal warning in >> > /usr/local/bro/share/bro/policy/frameworks/intel/seen/__load__.bro, line >> > 1: Discarded extraneous Broxygen comment: check link in mail_links >> > internal warning in >> > /usr/local/bro/share/bro/policy/frameworks/intel/seen/__load__.bro, line >> > 1: Discarded extraneous Broxygen comment: for >> > internal warning in >> > /usr/local/bro/share/bro/policy/frameworks/intel/seen/__load__.bro, line >> > 1: Discarded extraneous Broxygen comment: print fmt ("log_mine >> > Log_mime: %s", rec); >> > internal warning in >> > /usr/local/bro/share/bro/policy/frameworks/intel/seen/__load__.bro, line >> > 1: Discarded extraneous Broxygen comment: aashish: need to port to file >> > analysis framework >> > internal warning in >> > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. >> > /conn-established.bro, >> > line 1: Discarded extraneous Broxygen comment: check link in mail_links >> > internal warning in >> > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. >> > /conn-established.bro, >> > line 1: Discarded extraneous Broxygen comment: for >> > internal warning in >> > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. >> > /conn-established.bro, >> > line 1: Discarded extraneous Broxygen comment: print fmt >> > ("log_mine Log_mime: %s", rec); >> > internal warning in >> > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. >> > /conn-established.bro, >> > line 1: Discarded extraneous Broxygen comment: aashish: need to port to >> > file analysis framework >> > internal warning in >> > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. >> > /conn-established.bro, >> > line 1: Discarded extraneous Broxygen comment: check link in mail_links >> > internal warning in >> > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. >> > /conn-established.bro, >> > line 1: Discarded extraneous Broxygen comment: for >> > internal warning in >> > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. >> > /conn-established.bro, >> > line 1: Discarded extraneous Broxygen comment: print fmt >> > ("log_mine Log_mime: %s", rec); >> > internal warning in >> > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. >> > /conn-established.bro, >> > line 1: Discarded extraneous Broxygen comment: aashish: need to port to >> > file analysis framework >> > internal warning in >> > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. >> > /./where-locations.bro, >> > line 1: Discarded extraneous Broxygen comment: check link in mail_links >> > internal warning in >> > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. >> > /./where-locations.bro, >> > line 1: Discarded extraneous Broxygen comment: for >> > internal warning in >> > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. >> > /./where-locations.bro, >> > line 1: Discarded extraneous Broxygen comment: print fmt >> > ("log_mine Log_mime: %s", rec); >> > internal warning in >> > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. >> > /./where-locations.bro, >> > line 1: Discarded extraneous Broxygen comment: aashish: need to port to >> > file analysis framework >> > internal warning in >> > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. >> > /./where-locations.bro, >> > line 1: Discarded extraneous Broxygen comment: check link in mail_links >> > internal warning in >> > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. >> > /./where-locations.bro, >> > line 1: Discarded extraneous Broxygen comment: for >> > internal warning in >> > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. >> > /./where-locations.bro, >> > line 1: Discarded extraneous Broxygen comment: print fmt >> > ("log_mine Log_mime: %s", rec); >> > internal warning in >> > /usr/local/bro/share/bro/policy/frameworks/intel/seen/. >> > /./where-locations.bro, >> > line 1: Discarded extraneous Broxygen comment: aashish: need to port to >> > file analysis framework >> > , line 1: listening on eth0 >> > >> > , line 1: listening on eth1 >> > >> > 1503589314.254774 error in , line 1: Bad IP address: 5 >> > 1503589314.254774 error in , line 1: Bad IP address: 6 >> > 1503589314.254774 error in , line 1: Bad IP address: 1 >> > >> > Anything I need to be concerned about here? Thank you. >> > >> > James >> > _______________________________________________ >> > Bro mailing list >> > bro at bro-ids.org >> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From dnj0496 at gmail.com Thu Aug 24 18:52:27 2017 From: dnj0496 at gmail.com (Dk Jack) Date: Thu, 24 Aug 2017 18:52:27 -0700 Subject: [Bro] debug prints Message-ID: Hi All, I am noticing a weird issue. I am not seeing debug prints from my scripts. Initially some debug prints show up in stdout.log. Debug prints I make when analyzing traffic i.e. http_header event etc are not showing up. I am using flush_all() in my scripts and yet the debugs are not showing up in stdout.log. Could someone comment on what could be going here? Thanks. Dk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170824/3092a80e/attachment.html From dnj0496 at gmail.com Thu Aug 24 18:55:36 2017 From: dnj0496 at gmail.com (Dk Jack) Date: Thu, 24 Aug 2017 18:55:36 -0700 Subject: [Bro] http multi-part Message-ID: Hi, I am trying to perform some analysis on the HTTP body. For regular messages I am accumulating the http body using http_entity_data and http_end_entity events. However, this doesn't seem to work for multi-part post message. How do I accumulate multi-part post messages. Any help is appreciated. Thanks. Dk. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170824/dfc5f631/attachment.html From pssunu6 at gmail.com Thu Aug 24 23:39:15 2017 From: pssunu6 at gmail.com (ps sunu) Date: Fri, 25 Aug 2017 12:09:15 +0530 Subject: [Bro] binpac Message-ID: Hi, Hi how to write protocol using binPac ? any working sample available ? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170825/fca52914/attachment.html From vladg at illinois.edu Fri Aug 25 10:42:54 2017 From: vladg at illinois.edu (Vlad Grigorescu) Date: Fri, 25 Aug 2017 12:42:54 -0500 Subject: [Bro] binpac In-Reply-To: <201708250643.v7P6hYuH018474@vladg.net> References: <201708250643.v7P6hYuH018474@vladg.net> Message-ID: I might start with this tutorial: https://www.bro.org/development/howtos/binpac-sample-analyzer.html There's also this presentation I gave a few BroCons back: https://www.youtube.com/watch?v=1eDIl9y6ZnM --Vlad ps sunu writes: > Hi, > Hi how to write protocol using binPac ? any working sample available ? > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 800 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170825/ddb91232/attachment.bin From jmellander at lbl.gov Fri Aug 25 11:21:42 2017 From: jmellander at lbl.gov (Jim Mellander) Date: Fri, 25 Aug 2017 11:21:42 -0700 Subject: [Bro] http multi-part In-Reply-To: References: Message-ID: Hi: The attached policy should help you. It assembles multipart HTTP POSTs, and performs regular expression matching on the POST contents. Jim On Thu, Aug 24, 2017 at 6:55 PM, Dk Jack wrote: > Hi, > I am trying to perform some analysis on the HTTP body. For regular > messages I am accumulating the http body using http_entity_data > and http_end_entity events. However, this doesn't seem to work for > multi-part post message. How do I accumulate multi-part post messages. Any > help is appreciated. Thanks. > > Dk. > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170825/7e25580e/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: http-sensitive_POSTs.bro Type: application/octet-stream Size: 2839 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170825/7e25580e/attachment-0001.obj From dnj0496 at gmail.com Fri Aug 25 16:52:12 2017 From: dnj0496 at gmail.com (Dk Jack) Date: Fri, 25 Aug 2017 16:52:12 -0700 Subject: [Bro] http multi-part In-Reply-To: References: Message-ID: Hi Jim, I am doing something very similar to what you are doing in your script. However, I am unable to capture the body when the content-type is: Content-Type: multipart/form-data; boundary=----WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n When I use entity_data and end_entity event functions, what I accumulate for body when I receive a multi-part body is just a small string: YWFhYWFh However, I know my body is much larger. I've copy pasted the wireshark output of the multi-part content to the end of this email. I see there are mime_xxx functions. Should I be using them to capture multi-part content. If they should be used, any help in how they should be used would be most appreciated. Thanks. Dk. [HTTP request 1/1] File Data: 736 bytes MIME Multipart Media Encapsulation, Type: multipart/form-data, Boundary: "---- WebKitFormBoundarygsgnAl2Dz3rduY2p" [Type: multipart/form-data] First boundary: ------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n Encapsulated multipart part: Content-Disposition: form-data; name="success_url"\r\n\r\n Boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n Encapsulated multipart part: Content-Disposition: form-data; name="error_url"\r\n\r\n Boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n Encapsulated multipart part: Content-Disposition: form-data; name="firstname"\r\n\r\n Data (1 byte) 0000 61 a Data: 61 [Length: 1] Boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n Encapsulated multipart part: Content-Disposition: form-data; name="lastname"\r\n\r\n Data (1 byte) 0000 62 b Data: 62 [Length: 1] Boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n Encapsulated multipart part: Content-Disposition: form-data; name="email"\r\n\r\n Data (7 bytes) 0000 62 40 63 2e 63 6f 6d b at c.com Data: 6240632e636f6d [Length: 7] Boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n Encapsulated multipart part: Content-Disposition: form-data; name="password"\r\n\r\n Data (6 bytes) 0000 61 61 61 61 61 61 aaaaaa Data: 616161616161 [Length: 6] Boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n Encapsulated multipart part: Content-Disposition: form-data; name="confirmation"\r\n\r\n Data (6 bytes) 0000 61 61 61 61 61 61 aaaaaa Data: 616161616161 [Length: 6] Last boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p--\r\n On Fri, Aug 25, 2017 at 11:21 AM, Jim Mellander wrote: > Hi: > > The attached policy should help you. It assembles multipart HTTP POSTs, > and performs regular expression matching on the POST contents. > > Jim > > > On Thu, Aug 24, 2017 at 6:55 PM, Dk Jack wrote: > >> Hi, >> I am trying to perform some analysis on the HTTP body. For regular >> messages I am accumulating the http body using http_entity_data >> and http_end_entity events. However, this doesn't seem to work for >> multi-part post message. How do I accumulate multi-part post messages. Any >> help is appreciated. Thanks. >> >> Dk. >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170825/493c3a47/attachment.html From matiasdavaro at gmail.com Sat Aug 26 05:40:33 2017 From: matiasdavaro at gmail.com (Matias Davaro) Date: Sat, 26 Aug 2017 08:40:33 -0400 Subject: [Bro] binpac In-Reply-To: References: Message-ID: There are also these tutorials by opensm: https://www.youtube.com/watch?v=eZAgqSFd9-c https://www.youtube.com/watch?v=o2XdMdyJHzk On Fri, Aug 25, 2017 at 2:39 AM, ps sunu wrote: > Hi, > Hi how to write protocol using binPac ? any > working sample available ? > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170826/dbc39c2a/attachment.html From jmellander at lbl.gov Sat Aug 26 10:41:02 2017 From: jmellander at lbl.gov (Jim Mellander) Date: Sat, 26 Aug 2017 10:41:02 -0700 Subject: [Bro] http multi-part In-Reply-To: References: Message-ID: Perhaps posting the script you are using and a sample pcap might help to debug the issue you are having. Cheers, Jim On Fri, Aug 25, 2017 at 4:52 PM, Dk Jack wrote: > Hi Jim, > I am doing something very similar to what you are doing in your script. > However, I am unable to capture the body when the content-type is: > > Content-Type: multipart/form-data; boundary=---- > WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n > > When I use entity_data and end_entity event functions, what I accumulate > for body when I receive a multi-part body is just a small string: > > YWFhYWFh > > However, I know my body is much larger. I've copy pasted the wireshark > output of the multi-part content to the end of this email. > > I see there are mime_xxx functions. Should I be using them to capture > multi-part content. If they should be used, any help in how they should be > used would be most appreciated. Thanks. > > Dk. > > [HTTP request 1/1] > File Data: 736 bytes > MIME Multipart Media Encapsulation, Type: multipart/form-data, Boundary: > "---- WebKitFormBoundarygsgnAl2Dz3rduY2p" > [Type: multipart/form-data] > First boundary: ------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n > Encapsulated multipart part: > Content-Disposition: form-data; name="success_url"\r\n\r\n > Boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n > Encapsulated multipart part: > Content-Disposition: form-data; name="error_url"\r\n\r\n > Boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n > Encapsulated multipart part: > Content-Disposition: form-data; name="firstname"\r\n\r\n > Data (1 byte) > > 0000 61 a > Data: 61 > [Length: 1] > Boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n > Encapsulated multipart part: > Content-Disposition: form-data; name="lastname"\r\n\r\n > Data (1 byte) > > 0000 62 b > Data: 62 > [Length: 1] > Boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n > Encapsulated multipart part: > Content-Disposition: form-data; name="email"\r\n\r\n > Data (7 bytes) > > 0000 62 40 63 2e 63 6f 6d b at c.com > Data: 6240632e636f6d > [Length: 7] > Boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n > Encapsulated multipart part: > Content-Disposition: form-data; name="password"\r\n\r\n > Data (6 bytes) > > 0000 61 61 61 61 61 61 aaaaaa > Data: 616161616161 > [Length: 6] > Boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n > Encapsulated multipart part: > Content-Disposition: form-data; name="confirmation"\r\n\r\n > Data (6 bytes) > > 0000 61 61 61 61 61 61 aaaaaa > Data: 616161616161 > [Length: 6] > Last boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p--\r\n > > > On Fri, Aug 25, 2017 at 11:21 AM, Jim Mellander > wrote: > >> Hi: >> >> The attached policy should help you. It assembles multipart HTTP POSTs, >> and performs regular expression matching on the POST contents. >> >> Jim >> >> >> On Thu, Aug 24, 2017 at 6:55 PM, Dk Jack wrote: >> >>> Hi, >>> I am trying to perform some analysis on the HTTP body. For regular >>> messages I am accumulating the http body using http_entity_data >>> and http_end_entity events. However, this doesn't seem to work for >>> multi-part post message. How do I accumulate multi-part post messages. Any >>> help is appreciated. Thanks. >>> >>> Dk. >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170826/1474c567/attachment-0001.html From dnj0496 at gmail.com Mon Aug 28 11:13:30 2017 From: dnj0496 at gmail.com (Dk Jack) Date: Mon, 28 Aug 2017 11:13:30 -0700 Subject: [Bro] http multi-part In-Reply-To: References: Message-ID: I did some more investigation and it seems like my understanding how bro works was incomplete. What I observed is, for each entity/boundary in a multipart form request, Bro is raising a unique pair http_entity_data and http_end_entity events. For example, if you send a curl request as shown below: curl -v --form "key1=value1" --form "key2=value2", --form "key3=value3" then bro is raising three pairs of http_entity_data and http_end_entity events. I was assuming I'd just get one pair of events. Since I was expecting only one event, I was over writing the body I had saved from the previous event. As a result, my body variable was retaining only the last part of the multi-part form. On Sat, Aug 26, 2017 at 10:41 AM, Jim Mellander wrote: > Perhaps posting the script you are using and a sample pcap might help to > debug the issue you are having. > > Cheers, > > Jim > > On Fri, Aug 25, 2017 at 4:52 PM, Dk Jack wrote: > >> Hi Jim, >> I am doing something very similar to what you are doing in your script. >> However, I am unable to capture the body when the content-type is: >> >> Content-Type: multipart/form-data; boundary=----WebKitFormBoundar >> ygsgnAl2Dz3rduY2p\r\n >> >> When I use entity_data and end_entity event functions, what I accumulate >> for body when I receive a multi-part body is just a small string: >> >> YWFhYWFh >> >> However, I know my body is much larger. I've copy pasted the wireshark >> output of the multi-part content to the end of this email. >> >> I see there are mime_xxx functions. Should I be using them to capture >> multi-part content. If they should be used, any help in how they should be >> used would be most appreciated. Thanks. >> >> Dk. >> >> [HTTP request 1/1] >> File Data: 736 bytes >> MIME Multipart Media Encapsulation, Type: multipart/form-data, Boundary: >> "---- WebKitFormBoundarygsgnAl2Dz3rduY2p" >> [Type: multipart/form-data] >> First boundary: ------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n >> Encapsulated multipart part: >> Content-Disposition: form-data; name="success_url"\r\n\r\n >> Boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n >> Encapsulated multipart part: >> Content-Disposition: form-data; name="error_url"\r\n\r\n >> Boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n >> Encapsulated multipart part: >> Content-Disposition: form-data; name="firstname"\r\n\r\n >> Data (1 byte) >> >> 0000 61 a >> Data: 61 >> [Length: 1] >> Boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n >> Encapsulated multipart part: >> Content-Disposition: form-data; name="lastname"\r\n\r\n >> Data (1 byte) >> >> 0000 62 b >> Data: 62 >> [Length: 1] >> Boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n >> Encapsulated multipart part: >> Content-Disposition: form-data; name="email"\r\n\r\n >> Data (7 bytes) >> >> 0000 62 40 63 2e 63 6f 6d b at c.com >> Data: 6240632e636f6d >> [Length: 7] >> Boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n >> Encapsulated multipart part: >> Content-Disposition: form-data; name="password"\r\n\r\n >> Data (6 bytes) >> >> 0000 61 61 61 61 61 61 aaaaaa >> Data: 616161616161 >> [Length: 6] >> Boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p\r\n >> Encapsulated multipart part: >> Content-Disposition: form-data; name="confirmation"\r\n\r\n >> Data (6 bytes) >> >> 0000 61 61 61 61 61 61 aaaaaa >> Data: 616161616161 >> [Length: 6] >> Last boundary: \r\n------WebKitFormBoundarygsgnAl2Dz3rduY2p--\r\n >> >> >> On Fri, Aug 25, 2017 at 11:21 AM, Jim Mellander >> wrote: >> >>> Hi: >>> >>> The attached policy should help you. It assembles multipart HTTP POSTs, >>> and performs regular expression matching on the POST contents. >>> >>> Jim >>> >>> >>> On Thu, Aug 24, 2017 at 6:55 PM, Dk Jack wrote: >>> >>>> Hi, >>>> I am trying to perform some analysis on the HTTP body. For regular >>>> messages I am accumulating the http body using http_entity_data >>>> and http_end_entity events. However, this doesn't seem to work for >>>> multi-part post message. How do I accumulate multi-part post messages. Any >>>> help is appreciated. Thanks. >>>> >>>> Dk. >>>> >>>> _______________________________________________ >>>> Bro mailing list >>>> bro at bro-ids.org >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170828/7320d3fb/attachment.html From jdickenson at gmail.com Tue Aug 29 10:54:56 2017 From: jdickenson at gmail.com (James Dickenson) Date: Tue, 29 Aug 2017 10:54:56 -0700 Subject: [Bro] Question: using Log Filter Framework Message-ID: Hey All, Hey I've been trying to use the log filter framework to filter out various things like certain IP ranges and other records. Here is my attempt but Bro is complaining about line 8 saying my syntax isn't correct. Does anyone know what I'm doing wrong? I've been using Seth's blog post on the framework as an example of how to build the log filters. ---> http://blog.bro.org/2012/02/filtering-logs-with-bro.html Thanks in advance! -James D. ###### code ####### global filter_ip_set: subnet = 1.0.0.0/8 &redef; event bro_init() { Log::remove_default_filter(HTTP::LOG); Log::add_filter(HTTP::LOG, [$name="orig_h_filter", $pred(rec: HTTP::Info) = { rec$id$orig_h in $filter_ip_set; } ]); Log::add_filter(HTTP::LOG, [$name="resp_h_filter", $pred(rec: HTTP::Info) = { rec$id$resp_h in $filter_ip_set; } ]); } ######################### ###### error ######## error in **snip*** test.bro, line 8: syntax error, at or near ";" From jazoff at illinois.edu Tue Aug 29 11:14:38 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 29 Aug 2017 18:14:38 +0000 Subject: [Bro] Question: using Log Filter Framework In-Reply-To: References: Message-ID: > On Aug 29, 2017, at 1:54 PM, James Dickenson wrote: > > Hey All, > > Hey I've been trying to use the log filter framework to filter out > various things like certain IP ranges and other records. Here is my > attempt but Bro is complaining about line 8 saying my syntax isn't > correct. Does anyone know what I'm doing wrong? I've been using > Seth's blog post on the framework as an example of how to build the > log filters. ---> > http://blog.bro.org/2012/02/filtering-logs-with-bro.html > > > Thanks in advance! > > -James D. > > > ###### code ####### > > global filter_ip_set: subnet = 1.0.0.0/8 &redef; > > event bro_init() > { > Log::remove_default_filter(HTTP::LOG); > Log::add_filter(HTTP::LOG, [$name="orig_h_filter", > $pred(rec: HTTP::Info) = { > rec$id$orig_h in $filter_ip_set; > } ]); > Log::add_filter(HTTP::LOG, [$name="resp_h_filter", > $pred(rec: HTTP::Info) = { > rec$id$resp_h in $filter_ip_set; > } ]); > } > > ######################### > > ###### error ######## > error in **snip*** test.bro, line 8: syntax error, at or near ";" That just has 2 small issues: $filter_ip_set should be filter_ip_set and the pred function needs to return a boolean, so: Log::add_filter(HTTP::LOG, [$name="orig_h_filter", $pred(rec: HTTP::Info) = { return rec$id$orig_h in filter_ip_set; } ]); Personally I'd format that sort of thing like Log::add_filter(HTTP::LOG, [ $name="orig_h_filter", $pred(rec: HTTP::Info) = { return rec$id$orig_h in filter_ip_set; } ]); Also, not sure if that is just an example, but you'd probably want something like Log::add_filter(HTTP::LOG, [ $name="host_filter", $pred(rec: HTTP::Info) = { return rec$id$orig_h in filter_ip_set || rec$id$resp_h in filter_ip_set; } ]); Finally.. if you are doing more complicated things in the pred function splitting the statement up can help readability. function interesting_connections(rec: HTTP::Info): bool { return rec$id$orig_h in filter_ip_set || rec$id$resp_h in filter_ip_set; } event bro_init() { local filter: Log::Filter = [$name="host_filter", $pred=interesting_connections]; Log::add_filter(HTTP::LOG, filter); } -- - Justin Azoff From jdickenson at gmail.com Tue Aug 29 13:31:06 2017 From: jdickenson at gmail.com (James Dickenson) Date: Tue, 29 Aug 2017 13:31:06 -0700 Subject: [Bro] Question: using Log Filter Framework In-Reply-To: References: Message-ID: Thanks Justin! That did the trick, now I can start getting myself into deeper trouble! -James On Tue, Aug 29, 2017 at 11:14 AM, Azoff, Justin S wrote: > >> On Aug 29, 2017, at 1:54 PM, James Dickenson wrote: >> >> Hey All, >> >> Hey I've been trying to use the log filter framework to filter out >> various things like certain IP ranges and other records. Here is my >> attempt but Bro is complaining about line 8 saying my syntax isn't >> correct. Does anyone know what I'm doing wrong? I've been using >> Seth's blog post on the framework as an example of how to build the >> log filters. ---> >> http://blog.bro.org/2012/02/filtering-logs-with-bro.html >> >> >> Thanks in advance! >> >> -James D. >> >> >> ###### code ####### >> >> global filter_ip_set: subnet = 1.0.0.0/8 &redef; >> >> event bro_init() >> { >> Log::remove_default_filter(HTTP::LOG); >> Log::add_filter(HTTP::LOG, [$name="orig_h_filter", >> $pred(rec: HTTP::Info) = { >> rec$id$orig_h in $filter_ip_set; >> } ]); >> Log::add_filter(HTTP::LOG, [$name="resp_h_filter", >> $pred(rec: HTTP::Info) = { >> rec$id$resp_h in $filter_ip_set; >> } ]); >> } >> >> ######################### >> >> ###### error ######## >> error in **snip*** test.bro, line 8: syntax error, at or near ";" > > That just has 2 small issues: > > $filter_ip_set should be filter_ip_set and the pred function needs to return a boolean, so: > > Log::add_filter(HTTP::LOG, [$name="orig_h_filter", > $pred(rec: HTTP::Info) = { > return rec$id$orig_h in filter_ip_set; > } ]); > > Personally I'd format that sort of thing like > > Log::add_filter(HTTP::LOG, [ > $name="orig_h_filter", > $pred(rec: HTTP::Info) = { > return rec$id$orig_h in filter_ip_set; > } > ]); > > Also, not sure if that is just an example, but you'd probably want something like > > Log::add_filter(HTTP::LOG, [ > $name="host_filter", > $pred(rec: HTTP::Info) = { > return rec$id$orig_h in filter_ip_set || rec$id$resp_h in filter_ip_set; > } > ]); > > > Finally.. if you are doing more complicated things in the pred function splitting the statement up can help readability. > > function interesting_connections(rec: HTTP::Info): bool > { > return rec$id$orig_h in filter_ip_set || rec$id$resp_h in filter_ip_set; > } > > event bro_init() > { > local filter: Log::Filter = [$name="host_filter", $pred=interesting_connections]; > Log::add_filter(HTTP::LOG, filter); > } > > > > > -- > - Justin Azoff From brianallen at wustl.edu Wed Aug 30 15:38:49 2017 From: brianallen at wustl.edu (Allen, Brian) Date: Wed, 30 Aug 2017 22:38:49 +0000 Subject: [Bro] broctl startup error Message-ID: Hi, I just upgraded my BRO cluster machines from ubuntu 14.02 to 16.04. I just installed the latest version of BRO on the manager. All the machines in the cluster return this: $ bro --version bro version 2.5-294 But when I try to run broctl on the manager to start it up again, I get this error: $ sudo broctl Warning: the SitePolicyStandalone option is deprecated (use SitePolicyScripts instead). Welcome to BroControl 1.7-7 Type "help" for help. [BroControl] > check manager scripts failed. /bin/sh: 1: Syntax error: "(" unexpected proxy-1 scripts failed. /bin/sh: 1: Syntax error: "(" unexpected proxy-2 scripts failed. /bin/sh: 1: Syntax error: "(? unexpected . . .More of the same type of error . . . worker-3-16 scripts failed. /bin/sh: 1: Syntax error: "(" unexpected worker-3-17 scripts failed. /bin/sh: 1: Syntax error: "(" unexpected worker-3-18 scripts failed. /bin/sh: 1: Syntax error: "(" unexpected Any suggestions for where to find the file/script with the problem? Thanks for your help, -Brian ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170830/07a5f022/attachment.html From jazoff at illinois.edu Wed Aug 30 17:46:53 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 31 Aug 2017 00:46:53 +0000 Subject: [Bro] broctl startup error In-Reply-To: References: Message-ID: <45BCF154-AF9D-4B38-A782-0C80A89017B4@illinois.edu> > On Aug 30, 2017, at 6:38 PM, Allen, Brian wrote: > > Hi, > I just upgraded my BRO cluster machines from ubuntu 14.02 to 16.04. I just installed the latest version of BRO on the manager. All the machines in the cluster return this: > > $ bro --version > bro version 2.5-294 > > > But when I try to run broctl on the manager to start it up again, I get this error: > > $ sudo broctl > Warning: the SitePolicyStandalone option is deprecated (use SitePolicyScripts instead). > > Welcome to BroControl 1.7-7 > > Type "help" for help. > > [BroControl] > check > manager scripts failed. > /bin/sh: 1: Syntax error: "(" unexpected Well, that's a new one.. I wonder if you have a '(' in one of your settings.. Does broctl config | grep '(' or grep '^[^#].*(' -r /usr/local/bro/etc/ return anything? I was able to reproduce a similar error by adding env_vars=foo=(bar to the config. -- - Justin Azoff From brianallen at wustl.edu Thu Aug 31 05:58:55 2017 From: brianallen at wustl.edu (Allen, Brian) Date: Thu, 31 Aug 2017 12:58:55 +0000 Subject: [Bro] broctl startup error In-Reply-To: <45BCF154-AF9D-4B38-A782-0C80A89017B4@illinois.edu> References: <45BCF154-AF9D-4B38-A782-0C80A89017B4@illinois.edu> Message-ID: That helped! In broctl.cfg I had to fix this line. The single quotes were missing. BroArgs = -f '(net 128.252.0.0/16 or net 65.254.96.0/19)? But now when I start up the cluster (and it does start up which is good) the workers are all running at 100% which is not normal on these boxes. Should be around 50% That seems like pf_ring is not running, but I keep checking and pf_ring is installed and loaded. Hyperthreading is still disabled. That didn?t change after the upgrade. What could have changed after the upgrade to cause the cpus to run at 100%? I still think there is something wrong with pf_ring, but I?m not seeing it. I just built another BRO cluster for our research network on ubuntu 14.04 servers and got pf_ring and bro running there no problem. Thanks for your help, -Brian From: "Azoff, Justin S" > Date: Wednesday, August 30, 2017 at 7:46 PM To: Brian Allen > Cc: Bro-Mailinglist > Subject: Re: [Bro] broctl startup error On Aug 30, 2017, at 6:38 PM, Allen, Brian > wrote: Hi, I just upgraded my BRO cluster machines from ubuntu 14.02 to 16.04. I just installed the latest version of BRO on the manager. All the machines in the cluster return this: $ bro --version bro version 2.5-294 But when I try to run broctl on the manager to start it up again, I get this error: $ sudo broctl Warning: the SitePolicyStandalone option is deprecated (use SitePolicyScripts instead). Welcome to BroControl 1.7-7 Type "help" for help. [BroControl] > check manager scripts failed. /bin/sh: 1: Syntax error: "(" unexpected Well, that's a new one.. I wonder if you have a '(' in one of your settings.. Does broctl config | grep '(' or grep '^[^#].*(' -r /usr/local/bro/etc/ return anything? I was able to reproduce a similar error by adding env_vars=foo=(bar to the config. -- - Justin Azoff ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170831/6eadfac2/attachment-0001.html From jazoff at illinois.edu Thu Aug 31 06:22:12 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 31 Aug 2017 13:22:12 +0000 Subject: [Bro] broctl startup error In-Reply-To: References: <45BCF154-AF9D-4B38-A782-0C80A89017B4@illinois.edu> Message-ID: On Aug 31, 2017, at 8:58 AM, Allen, Brian wrote: > > That helped! In broctl.cfg I had to fix this line. The single quotes were missing. > > BroArgs = -f '(net 128.252.0.0/16 or net 65.254.96.0/19)? > > But now when I start up the cluster (and it does start up which is good) the workers are all running at 100% which is not normal on these boxes. Should be around 50% That seems like pf_ring is not running, but I keep checking and pf_ring is installed and loaded. > > Hyperthreading is still disabled. That didn?t change after the upgrade. > > What could have changed after the upgrade to cause the cpus to run at 100%? I still think there is something wrong with pf_ring, but I?m not seeing it. I just built another BRO cluster for our research network on ubuntu 14.04 servers and got pf_ring and bro running there no problem. > > Thanks for your help, > -Brian Yes.. it's likely that bro is not using pf_ring properly. What does this output? broctl exec "ldd `which bro`|grep pcap" You should see lines like libpcap.so.1 => /opt/pfring/lib/libpcap.so.1 and not the normal libpcap in /usr/lib If you look at your conn.log you also may see the same exact connection logged once for each worker that you are running. -- - Justin Azoff From gl89 at cornell.edu Thu Aug 31 08:39:23 2017 From: gl89 at cornell.edu (Glenn Forbes Fleming Larratt) Date: Thu, 31 Aug 2017 11:39:23 -0400 (EDT) Subject: [Bro] Nested modules in /opt/app/bro/share/bro/site/ not working? Message-ID: Folks, This may well be an RTFM - I just don't know which section would cover this. I have two modules created in /opt/app/bro/share/bro/site/ , with the intention of having one @load the other - but I cannot seem to get the combination to work, and I'm unsure even where the failure is. The @load'ed module, "CU_net_defs", is intended as a shim to some .csv files, periodically updated from the central DB which tracks internal subnet allocations, which is supposed to instantiate two global sets of CIDR's, one for top-level allocations (e.g. 128.253.0.0), and the other for internally-allocated subnets (e.g. 128.253.101.0/25). The @load'ing module, "bro-wsSMTP", is intended to detect user workstations et.al. that have been compromised and are acting as spambots, by the following methodology. - understand the top-level allocations by @load'ing CU_net_defs; - understand an internal set of our e-mail infrastructure hosts and networks; and - generate logfiles, "ws-smtp.{}.log", containing entries where -- id$orig_h is part of a campus allocation; -- id$orig_h is *not* part of our e-mail infrastructure; and -- id$resp_h is *not* part of a campus allocation. The final goal is to have the resulting "ws-smtp.{}.log" files fed into Splunk for detection/correlation. The thing is, once I create an expression intended to isolate by way of the criteria above, I get no results whatever - even though the data is clearly in the normal smtp.log file. Slightly abridged instances of the module files are appended below. Can anyone shed any light on this? Thanks for any info, -- Glenn Forbes Fleming Larratt Cornell University IT Security Office ===== CU_net_defs/CU_net_defs.bro ===== module CU_net_defs; export { type IDX: record { cidr: subnet; }; global CU_subnets: set[subnet]; global CU_allocations: set[subnet]; } event bro_init() { Input::add_table([$source="/users/bro/devel/DNSDBnc/CU_subnets.txt", $name="CU_subnets", $idx = IDX, $destination=CU_subnets]); Input::remove("CU_subnets"); Input::add_table([$source="/users/bro/devel/DNSDBnc/CU_allocations.txt", $name="CU_allocations", $idx = IDX, $destination=CU_allocations]); Input::remove("CU_allocations"); } ======================================= ===== bro-wsSMTP/ws-SMTP-outbound.bro ===== @load base/protocols/smtp @load CU_net_defs module ws_SMTP_outbound; export { global local_mail_infra: set[subnet] = { 128.253.150.128/25, [stuff redacted...] 132.236.101.7/32 }; } event bro_init() { Log::add_filter(SMTP::LOG, [$name = "ws-smtp-outbound", $path = "ws-smtp", $pred(rec: SMTP::Info) = { return rec$id?$orig_h && rec$id$orig_h in CU_net_defs::CU_allocations && !(rec$id$orig_h in local_mail_infra) && rec$id?$resp_h && !(rec$id$resp_h in CU_net_defs::CU_allocations) ; } ] ); } =========================================== From jazoff at illinois.edu Thu Aug 31 08:58:51 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 31 Aug 2017 15:58:51 +0000 Subject: [Bro] Nested modules in /opt/app/bro/share/bro/site/ not working? In-Reply-To: References: Message-ID: > On Aug 31, 2017, at 11:39 AM, Glenn Forbes Fleming Larratt wrote: > > Folks, > > This may well be an RTFM - I just don't know which section would cover > this. > > I have two modules created in /opt/app/bro/share/bro/site/ , with the > intention of having one @load the other - but I cannot seem to get the > combination to work, and I'm unsure even where the failure is. Looks like you've done almost everything correctly! I think you just have some input files issues. > ... > ===== CU_net_defs/CU_net_defs.bro ===== > module CU_net_defs; > > export > { > type IDX: record { cidr: subnet; }; > > global CU_subnets: set[subnet]; > global CU_allocations: set[subnet]; > } > > event bro_init() > { > Input::add_table([$source="/users/bro/devel/DNSDBnc/CU_subnets.txt", $name="CU_subnets", > $idx = IDX, $destination=CU_subnets]); > Input::remove("CU_subnets"); > > > Input::add_table([$source="/users/bro/devel/DNSDBnc/CU_allocations.txt", $name="CU_allocations", > $idx = IDX, $destination=CU_allocations]); > Input::remove("CU_allocations"); > } I would remove the two Input::remove lines and add $mode=Input::REREAD to the two add_table calls. I'm not sure if the immediate remove would prevent it from working at all, but you don't want to do that anyway. You want the input files to stay active and also update the sets when the files are updated. Once you do that, check the reporter.log a few seconds after starting bro to see if it is complaining about accessing/parsing those 2 text files. Most likely the root cause of your problem is that something is preventing them from being parsed properly which is making your two sets empty which makes your smtp filter match nothing. If I had to guess, you are missing the required '#fields\tcidr' header. > ===== bro-wsSMTP/ws-SMTP-outbound.bro ===== > @load base/protocols/smtp > @load CU_net_defs > > > module ws_SMTP_outbound; > > export > { > global local_mail_infra: set[subnet] = > { > 128.253.150.128/25, > [stuff redacted...] > 132.236.101.7/32 > }; > } > > event bro_init() > { > Log::add_filter(SMTP::LOG, [$name = "ws-smtp-outbound", > $path = "ws-smtp", > $pred(rec: SMTP::Info) = > { > return rec$id?$orig_h > && rec$id$orig_h in CU_net_defs::CU_allocations > && !(rec$id$orig_h in local_mail_infra) > && rec$id?$resp_h > && !(rec$id$resp_h in CU_net_defs::CU_allocations) > ; > } > ] > ); > } unrelated to problems, but !(rec$id$orig_h in local_mail_infra) can be written as rec$id$orig_h !in local_mail_infra -- - Justin Azoff From dnj0496 at gmail.com Thu Aug 31 16:57:14 2017 From: dnj0496 at gmail.com (Dk Jack) Date: Thu, 31 Aug 2017 16:57:14 -0700 Subject: [Bro] http response timeout Message-ID: Hi, In my http.log, I am seeing some lines being written without response code etc. What could be the reason for this? One reason I could think of was, what if the server or some entity between bro and the server that dropped the request/response thus preventing the response from reaching bro or the connection is closed on receiving the request by a downstream security device. How does bro react in such cases? could one of these scenarios explain why the response fields are missing from the log? Dk. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170831/a5976b5f/attachment.html