From yuza.rasfar at gmail.com Mon Apr 3 00:09:37 2017 From: yuza.rasfar at gmail.com (tkg_cangkul) Date: Mon, 03 Apr 2017 14:09:37 +0700 Subject: [Bro] send all logs to kafka Message-ID: <58E1F531.1070403@gmail.com> hi, i'm trying to using bro kafka plugin to send the bro logs into kafka. i've a problem to send all the logs type to kafka. i've set this into my local.bro : *@load Bro/Kafka/logs-to-kafka.bro redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, CONN::LOG, Known::SERVICES_LOG, Weird::LOG, Notice::LOG); *but when i check on kafka topic. there are only *http, conn, & dns*. i've check in my bro logs dir and there are so many types of log. is there any config that i missed? pls help. Best Regards, Tukang_Cangkul -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170403/de09f47e/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot from 2017-04-03 14:08:41.png Type: image/png Size: 10553 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170403/de09f47e/attachment.bin From anastasakis62 at gmail.com Mon Apr 3 05:49:21 2017 From: anastasakis62 at gmail.com (mike anastasakis) Date: Mon, 3 Apr 2017 14:49:21 +0200 Subject: [Bro] TCP Conn Log Message-ID: Hello, I am using Bro for a project and I have a question regarding it's capabilities. Currently when I have a long TCP connection that includes frequent TCP Keep Alive messages, bro is reassembling the whole network trace into one connection and presents it in conn.log with a big duration value. Is it possible to make bro split up TCP connections into smaller fragments based on a interval I set up or at least whenever a TCP Keep alive handshake takes place? Regards, Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170403/87602be7/attachment.html From zeolla at gmail.com Mon Apr 3 06:03:02 2017 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Mon, 03 Apr 2017 13:03:02 +0000 Subject: [Bro] send all logs to kafka In-Reply-To: <58E1F531.1070403@gmail.com> References: <58E1F531.1070403@gmail.com> Message-ID: Are you sending all of those logs to the same topic? Some of your kafka-related bro configs are missing in the above email, can you send everything? For instance, Kafka::kafka_conf, Kafka::topic_name (if used), etc. How are you verifying that they are properly getting onto kafka? I've never sent anything other than http, conn, and dns to kafka before, but I feel like that should work. I could be wrong. Jon On Mon, Apr 3, 2017 at 3:17 AM tkg_cangkul wrote: > hi, > > i'm trying to using bro kafka plugin to send the bro logs into kafka. > i've a problem to send all the logs type to kafka. > > i've set this into my local.bro : > > > > > *@load Bro/Kafka/logs-to-kafka.bro redef Kafka::logs_to_send = > set(HTTP::LOG, DNS::LOG, CONN::LOG, Known::SERVICES_LOG, Weird::LOG, > Notice::LOG); *but when i check on kafka topic. there are only *http, > conn, & dns*. > i've check in my bro logs dir and there are so many types of log. > > > > is there any config that i missed? > pls help. > > Best Regards, > > Tukang_Cangkul > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170403/aa8df793/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot from 2017-04-03 14:08:41.png Type: image/png Size: 10553 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170403/aa8df793/attachment-0002.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot from 2017-04-03 14:08:41.png Type: image/png Size: 10553 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170403/aa8df793/attachment-0003.bin From yuza.rasfar at gmail.com Mon Apr 3 07:15:26 2017 From: yuza.rasfar at gmail.com (Youzha) Date: Mon, 03 Apr 2017 14:15:26 +0000 Subject: [Bro] send all logs to kafka In-Reply-To: References: <58E1F531.1070403@gmail.com> Message-ID: hi Zeolla, yeah i sending all the logs to the same topic (bro topic). maybe i do something wrong about the writing of config *set(HTTP::LOG, DNS::LOG, CONN::LOG, Known::SERVICES_LOG, Weird::LOG, Notice::LOG); ?* *maybe there are case sensitive words? or anything else? can you give me some lists of the logs that i can use? * On Mon, Apr 3, 2017 at 8:03 PM Zeolla at GMail.com wrote: > Are you sending all of those logs to the same topic? Some of your > kafka-related bro configs are missing in the above email, can you send > everything? For instance, Kafka::kafka_conf, Kafka::topic_name (if used), > etc. > > How are you verifying that they are properly getting onto kafka? I've > never sent anything other than http, conn, and dns to kafka before, but I > feel like that should work. I could be wrong. > > Jon > > On Mon, Apr 3, 2017 at 3:17 AM tkg_cangkul wrote: > > hi, > > i'm trying to using bro kafka plugin to send the bro logs into kafka. > i've a problem to send all the logs type to kafka. > > i've set this into my local.bro : > > > > > *@load Bro/Kafka/logs-to-kafka.bro redef Kafka::logs_to_send = > set(HTTP::LOG, DNS::LOG, CONN::LOG, Known::SERVICES_LOG, Weird::LOG, > Notice::LOG); *but when i check on kafka topic. there are only *http, > conn, & dns*. > i've check in my bro logs dir and there are so many types of log. > > > > is there any config that i missed? > pls help. > > Best Regards, > > Tukang_Cangkul > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -- > > Jon > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170403/7c82fabb/attachment.html From jazoff at illinois.edu Mon Apr 3 07:29:47 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 3 Apr 2017 14:29:47 +0000 Subject: [Bro] send all logs to kafka In-Reply-To: <58E1F531.1070403@gmail.com> References: <58E1F531.1070403@gmail.com> Message-ID: <32635C85-F77B-4543-8410-C57E7280C5EA@illinois.edu> > On Apr 3, 2017, at 3:09 AM, tkg_cangkul wrote: > > hi, > > i'm trying to using bro kafka plugin to send the bro logs into kafka. > i've a problem to send all the logs type to kafka. > > i've set this into my local.bro : > > @load Bro/Kafka/logs-to-kafka.bro > redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, CONN::LOG, Known::SERVICES_LOG, Weird::LOG, Notice::LOG); > > but when i check on kafka topic. there are only http, conn, & dns. > i've check in my bro logs dir and there are so many types of log. http,dns,conn are all high volume log files compared to known services, weird, and notice. Based on your file sizes it looks like you only had a few notice and known services log entries, so is it possible that you just missed them among the large volume of conn and dns log entries? Also, your weird log looks to be very large, you should do a cat weird.log |bro-cut name|sort|uniq -c|sort -nr|head -n 10 to see why you have so many weird entries. -- - Justin Azoff From yuza.rasfar at gmail.com Mon Apr 3 07:38:10 2017 From: yuza.rasfar at gmail.com (tkg_cangkul) Date: Mon, 03 Apr 2017 21:38:10 +0700 Subject: [Bro] send all logs to kafka In-Reply-To: <32635C85-F77B-4543-8410-C57E7280C5EA@illinois.edu> References: <58E1F531.1070403@gmail.com> <32635C85-F77B-4543-8410-C57E7280C5EA@illinois.edu> Message-ID: <58E25E52.6050206@gmail.com> Hi Azoff, I've running bro with that config about 2 days and the picture that i sent before is just a current log dir. This is the result of the command : cat weird.log |bro-cut name|sort|uniq -c|sort -nr|head -n 10 On 03/04/17 21:29, Azoff, Justin S wrote: >> On Apr 3, 2017, at 3:09 AM, tkg_cangkul wrote: >> >> hi, >> >> i'm trying to using bro kafka plugin to send the bro logs into kafka. >> i've a problem to send all the logs type to kafka. >> >> i've set this into my local.bro : >> >> @load Bro/Kafka/logs-to-kafka.bro >> redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, CONN::LOG, Known::SERVICES_LOG, Weird::LOG, Notice::LOG); >> >> but when i check on kafka topic. there are only http, conn, & dns. >> i've check in my bro logs dir and there are so many types of log. > > http,dns,conn are all high volume log files compared to known services, weird, and notice. > > Based on your file sizes it looks like you only had a few notice and known services log entries, so is it possible that you just missed them among the large volume of conn and dns log entries? > > Also, your weird log looks to be very large, you should do a > > cat weird.log |bro-cut name|sort|uniq -c|sort -nr|head -n 10 > > to see why you have so many weird entries. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170403/016bf108/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot from 2017-04-03 21:33:21.png Type: image/png Size: 9641 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170403/016bf108/attachment-0001.bin From yuza.rasfar at gmail.com Mon Apr 3 07:49:53 2017 From: yuza.rasfar at gmail.com (tkg_cangkul) Date: Mon, 03 Apr 2017 21:49:53 +0700 Subject: [Bro] send all logs to kafka In-Reply-To: References: <58E1F531.1070403@gmail.com> Message-ID: <58E26111.3090308@gmail.com> Sorry, i've missed to answer your question before . This is all of my config to bro-kafka . *@load Bro/Kafka/logs-to-kafka.bro redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, CONN::LOG, Known::ServicesInfo, Weird::LOG, Notice::LOG, SSH::LOG, SMTP::LOG, DHCP::LOG); redef Kafka::topic_name = "bro"; redef Kafka::tag_json = T; redef Kafka::kafka_conf = table(["metadata.broker.list"] = "hostname:6667"); *I can verifying that they are getting onto kafka or not by using this command : *bin/kafka-console-consumer.sh --bootstrap-server hostname:6667 --topic bro --from-beginning |grep weird* On 03/04/17 21:15, Youzha wrote: > hi Zeolla, > > yeah i sending all the logs to the same topic (bro topic). > > maybe i do something wrong about the writing of config *set(HTTP::LOG, > DNS::LOG, CONN::LOG, Known::SERVICES_LOG, Weird::LOG, Notice::LOG); ?* > * > * > *maybe there are case sensitive words? or anything else? can you give > me some lists of the logs that i can use? > * > > > On Mon, Apr 3, 2017 at 8:03 PM Zeolla at GMail.com > wrote: > > Are you sending all of those logs to the same topic? Some of your > kafka-related bro configs are missing in the above email, can you > send everything? For > instance, Kafka::kafka_conf, Kafka::topic_name (if used), etc. > > How are you verifying that they are properly getting onto kafka? > I've never sent anything other than http, conn, and dns to kafka > before, but I feel like that should work. I could be wrong. > > Jon > > On Mon, Apr 3, 2017 at 3:17 AM tkg_cangkul > wrote: > > hi, > > i'm trying to using bro kafka plugin to send the bro logs into > kafka. > i've a problem to send all the logs type to kafka. > > i've set this into my local.bro : > > *@load Bro/Kafka/logs-to-kafka.bro > redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, > CONN::LOG, Known::SERVICES_LOG, Weird::LOG, Notice::LOG); > > *but when i check on kafka topic. there are only *http, conn, > & dns*. > i've check in my bro logs dir and there are so many types of log. > > > > is there any config that i missed? > pls help. > > Best Regards, > > Tukang_Cangkul > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -- > > Jon > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170403/77232fda/attachment.html From jgin at utexas.edu Mon Apr 3 22:54:35 2017 From: jgin at utexas.edu (Jeremy Gin) Date: Tue, 4 Apr 2017 00:54:35 -0500 Subject: [Bro] Bro terminates on its own in PCAP read mode Message-ID: ? m_atk3_set0_t0.pcap ?Hello, I am trying to run Bro in PCAP read mode on PCAPs that contain flooding attacks created in a lab environment. I installed Bro from source and did not modify the local.bro. The command I am using is: "bro -r .pcap -C local --time" This returns the following output: "WARNING: No Site::local_nets have been defined. It's usually a good idea to define your local networks. # initialization 2.756138 # initialization 59M/49M Killed" I have attached the PCAP. My initial reaction is that the PCAP is too big as this happens to only PCAPs containing DOS attacks. However, the attached PCAP is 69 MB and Bro successfully runs on other PCAPs that are around 73 MB. Can anyone explain why Bro is terminating itself? Any insight you can provide is much appreciated. Thanks, Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170404/79523e83/attachment.html From bill.de.ping at gmail.com Tue Apr 4 03:07:25 2017 From: bill.de.ping at gmail.com (william de ping) Date: Tue, 4 Apr 2017 13:07:25 +0300 Subject: [Bro] minimalistic bro setup Message-ID: Hi all, I would like to make bro real thin by not loading all unnecessary plugins\analyzers. I have tweaked init-bare and init-default scripts, yet when I see the loaded-scripts, I see that many plugins are loaded. How can I turn off plugins effectively ? when I edit base/bif/plugins/__load__.bro to not load ,say, FTP, I get many errors that some FTP fields are not recognized and preventing the cluster from running. I basically need only UDP and DNS events and have no need for the moment for other down stream analyzers\plugins. Thanks in advance B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170404/433b0008/attachment.html From al.kefallonitis at gmail.com Tue Apr 4 08:31:10 2017 From: al.kefallonitis at gmail.com (Alex Kefallonitis) Date: Tue, 4 Apr 2017 18:31:10 +0300 Subject: [Bro] Virus Total Api Message-ID: I am using bro 2.5 and i cant get this working https://github.com/sooshie/bro-scripts/blob/master/misc/vt_check.bro I see curl running on request and succefully submitted on virustotal but i get this error: 1490780707.065084 error in /opt/bro/share/bro/bro-extra/vt_check.bro, line 79: no such index (VTCHECK::temp[2]) 1490780707.065084 error in /opt/bro/share/bro/bro-extra/vt_check.bro, line 74: no such index (VTCHECK::temp[2]) 1490780707.065084 error in /opt/bro/share/bro/bro-extra/vt_check.bro, line 91: value used but not set (VTCHECK::positives) Anyone made this work or anything similar? I can't get either this example working https://www.sans.org/reading-room/whitepapers/detection/detecting-malicious-smb-activity-bro-37472 Thanks in advanced -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170404/81b4ec44/attachment.html From jbarber at computer.org Tue Apr 4 09:24:14 2017 From: jbarber at computer.org (Jeff Barber) Date: Tue, 4 Apr 2017 10:24:14 -0600 Subject: [Bro] Bro terminates on its own in PCAP read mode In-Reply-To: References: Message-ID: Possibly just out of memory? That pcap has -- according to wireshark -- 678714 IPv6 conversations. So bro will create that many connection table entries. Those entries are not small and a number of related structures get created too, so you end up with a ton of memory used by bro. And the packets are all "received" within a few seconds so none of the connection table entries will have timed out by the time you get to the end. It's traditional on linux that the kernel allows memory to be "overcommitted" but then if the kernel runs out of memory for critical functions, it chooses a fat process to kill. References here: https://linux-mm.org/OOM_Killer https://unix.stackexchange.com/questions/153585/how-the-oom-killer-decides-which-process-to-kill-first So it's not that the pcap itself is too large -- bro basically reads and processes one packet at a time -- it's that processing it takes more memory than you have available. On Mon, Apr 3, 2017 at 11:54 PM, Jeremy Gin wrote: > ? > m_atk3_set0_t0.pcap > > ?Hello, > > I am trying to run Bro in PCAP read mode on PCAPs that contain flooding > attacks created in a lab environment. I installed Bro from source and did > not modify the local.bro. The command I am using is: > > "bro -r .pcap -C local --time" > > This returns the following output: > "WARNING: No Site::local_nets have been defined. It's usually a good idea > to define your local networks. > # initialization 2.756138 > # initialization 59M/49M > Killed" > > I have attached the PCAP. My initial reaction is that the PCAP is too big > as this happens to only PCAPs containing DOS attacks. However, the attached > PCAP is 69 MB and Bro successfully runs on other PCAPs that are around 73 > MB. Can anyone explain why Bro is terminating itself? > > Any insight you can provide is much appreciated. > > Thanks, > Jeremy > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170404/bf4e9879/attachment.html From jgin at utexas.edu Tue Apr 4 11:23:02 2017 From: jgin at utexas.edu (Jeremy Gin) Date: Tue, 4 Apr 2017 13:23:02 -0500 Subject: [Bro] Bro terminates on its own in PCAP read mode In-Reply-To: References: Message-ID: Thank you for explaining in-depth, Jeff. It does seem like Bro ran out of memory, but the VM I used to run Bro had 4 GB of RAM. I tried running it with 10 GB of RAM, and initially, it does seem to finish the process. If Bro is having such a hard time, how is this type of failure to be avoided in real life? Is taking down a Bro server really as simple as generating millions of conversations? Is this just a design flaw in Bro? On Tue, Apr 4, 2017 at 11:24 AM, Jeff Barber wrote: > Possibly just out of memory? That pcap has -- according to wireshark -- > 678714 IPv6 conversations. So bro will create that many connection table > entries. Those entries are not small and a number of related structures get > created too, so you end up with a ton of memory used by bro. And the > packets are all "received" within a few seconds so none of the connection > table entries will have timed out by the time you get to the end. > > It's traditional on linux that the kernel allows memory to be > "overcommitted" but then if the kernel runs out of memory for critical > functions, it chooses a fat process to kill. References here: > > https://linux-mm.org/OOM_Killer > https://unix.stackexchange.com/questions/153585/how-the- > oom-killer-decides-which-process-to-kill-first > > > So it's not that the pcap itself is too large -- bro basically reads and > processes one packet at a time -- it's that processing it takes more memory > than you have available. > > > On Mon, Apr 3, 2017 at 11:54 PM, Jeremy Gin wrote: > >> ? >> m_atk3_set0_t0.pcap >> >> ?Hello, >> >> I am trying to run Bro in PCAP read mode on PCAPs that contain flooding >> attacks created in a lab environment. I installed Bro from source and did >> not modify the local.bro. The command I am using is: >> >> "bro -r .pcap -C local --time" >> >> This returns the following output: >> "WARNING: No Site::local_nets have been defined. It's usually a good >> idea to define your local networks. >> # initialization 2.756138 >> # initialization 59M/49M >> Killed" >> >> I have attached the PCAP. My initial reaction is that the PCAP is too big >> as this happens to only PCAPs containing DOS attacks. However, the attached >> PCAP is 69 MB and Bro successfully runs on other PCAPs that are around 73 >> MB. Can anyone explain why Bro is terminating itself? >> >> Any insight you can provide is much appreciated. >> >> Thanks, >> Jeremy >> >> >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170404/da190daa/attachment-0001.html From kingsleyluoxin at hotmail.com Tue Apr 4 17:30:22 2017 From: kingsleyluoxin at hotmail.com (Luo Xin) Date: Wed, 5 Apr 2017 00:30:22 +0000 Subject: [Bro] How to implement state machine in bro? Message-ID: Recently, I have really been fascinated by the elegance of bro, and I have read some source codes of bro. Now I do want to add something to make bro stronger. With the increasing attention paid to anomaly detection, I would like to implement a specification based anomaly detection in bro. One of my available ideas is to implement protocol specification by means of protocol state machine. I do wonder how to accomplish that in bro. Is here anyone that has any idea or has done something similar before? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170405/bb5e7a8b/attachment.html From sbeaupied at salesforce.com Tue Apr 4 17:46:14 2017 From: sbeaupied at salesforce.com (Scott Beaupied) Date: Tue, 4 Apr 2017 20:46:14 -0400 Subject: [Bro] How to implement state machine in bro? In-Reply-To: References: Message-ID: What could really be used is a multi-thread manager. We're running into issues with "best practices" due to the single threading of the mgr and HW limits in our cluster. On Tue, Apr 4, 2017 at 8:30 PM, Luo Xin wrote: > Recently, I have really been fascinated by the elegance of bro, and I have > read some source codes of bro. Now I do want to add something to make bro > stronger. With the increasing attention paid to anomaly detection, I would > like to implement a specification based anomaly detection in bro. One of my > available ideas is to implement protocol specification by means of protocol > state machine. I do wonder how to accomplish that in bro. Is here anyone > that has any idea or has done something similar before? > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Scott Beaupied Senior Security DevOps Engineer, Pardot.com Salesforce.com, Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170404/6276f9a2/attachment.html From carlopmart at gmail.com Wed Apr 5 03:01:45 2017 From: carlopmart at gmail.com (C. L. Martinez) Date: Wed, 5 Apr 2017 10:01:45 +0000 Subject: [Bro] Yara integration with Bro 2.5 Message-ID: <20170405100145.c2slxlnnjbjtmy4y@scotland.uxdom.org> Hi all, Broyara works with Bro 2.5: https://github.com/hempnall/broyara?? Thanks -- Greetings, C. L. Martinez From pssunu6 at gmail.com Wed Apr 5 05:23:46 2017 From: pssunu6 at gmail.com (ps sunu) Date: Wed, 5 Apr 2017 17:53:46 +0530 Subject: [Bro] auth_bruteforcing.bro error Message-ID: I am using below code while running this i am getting below error from below area *if(!auth_success) {* * SumStats::observe("http.auth_errors.attacker",* * [$host=to_addr(c$http$cluster_client_ip)],* * []);* * if ( c?$conn )* error *field value missing [AuthBruteforcing::c$http$cluster_client_ip]* code @load base/frameworks/notice @load base/frameworks/sumstats @load base/protocols/http module AuthBruteforcing; export { redef enum Notice::Type += { ## Indicates that a host performing HTTP requests leading to ## excessive HTTP auth errors was detected. HTTP_AuthBruteforcing_Attacker, ## Indicates that a host was seen to respond excessive HTTP ## auth errors. This is tracked by IP address as opposed to ## hostname. HTTP_AuthBruteforcing_Victim, }; # Let's tag the http item redef enum HTTP::Tags += { ## HTTP status code 401, describing a HTTP auth error HTTP_AUTH_ERROR, ## HTTP describing a successful HTTP auth HTTP_AUTH_SUCCESS, }; redef enum Log::ID += { LOG }; type Info: record { ts: time &log; uid: string &log; id: conn_id &log &optional; cluster_client_ip: string &log &optional; status_code: count &log &optional; host: string &log &optional; uri: string &log &optional; username: string &log &optional; auth_success: bool &log &optional; }; global log_auth: event(rec: Info); ## Defines the threshold that determines if a auth bruteforcing attack ## is ongoing based on the number of requests that appear to be ## attacks. const auth_errors_threshold: double = 50.0 &redef; ## Interval at which to watch for the ## :bro:id:`AuthBruteforcing::auth_errors_requests_threshold` variable to be crossed. ## At the end of each interval the counter is reset. const auth_errors_interval = 5min &redef; ## Interval at which to watch for the ## :bro:id:`AuthBruteforcing::excessive_auth_errors_threshold` variable to be ## crossed. At the end of each interval the counter is reset. const excessive_auth_errors_interval = 1min &redef; const internal_space: subnet = 10.0.0.0/8 &redef; const public_space: subnet = 63.245.208.0/20 &redef; const ignore_host_resp: set[addr] = { } &redef; const ignore_host_orig: set[addr] = { } &redef; } event bro_init() &priority=3 { # Create auth_bruteforcing.log Log::create_stream(AuthBruteforcing::LOG, [$columns=Info, $ev=log_auth]); # HTTP auth errors for requests FROM the same host local r1: SumStats::Reducer = [$stream="http.auth_errors.attacker", $apply=set(SumStats::SUM)]; SumStats::create([$name="auth-http-errors-attackers", $epoch=auth_errors_interval, $reducers=set(r1), $threshold_val(key: SumStats::Key, result: SumStats::Result) = { return result["http.auth_errors.attacker"]$sum; }, $threshold=auth_errors_threshold, $threshold_crossed(key: SumStats::Key, result: SumStats::Result) = { NOTICE([$note=HTTP_AuthBruteforcing_Attacker, $msg=fmt("HTTP auth bruteforcing from attacker %s", key$host), $sub=fmt("%.0f auth failed in %s", result["http.auth_errors.attacker"]$sum, auth_errors_interval), $src=key$host, $n=to_count(fmt("%.0f", result["http.auth_errors.attacker"]$sum)) ]); }]); # HTTP errors for requests TO the same host local r2: SumStats::Reducer = [$stream="http.auth_errors.victim", $apply=set(SumStats::SUM)]; SumStats::create([$name="auth-http-errors-victims", $epoch=auth_errors_interval, $reducers=set(r2), $threshold_val(key: SumStats::Key, result: SumStats::Result) = { return result["http.auth_errors.victim"]$sum; }, $threshold=auth_errors_threshold, $threshold_crossed(key: SumStats::Key, result: SumStats::Result) = { NOTICE([$note=HTTP_AuthBruteforcing_Victim, $msg=fmt("HTTP auth bruteforcing to victim %s", key$host), $sub=fmt("%.0f auth failed in %s", result["http.auth_errors.victim"]$sum, auth_errors_interval), $src=key$host, $n=to_count(fmt("%.0f", result["http.auth_errors.victim"]$sum)) ]); }]); } # Make sure we have all the http info before looking for auth errors event http_message_done(c: connection, is_orig: bool, stat: http_message_stat) { # only conns we want local ports_ext: set[port] = { 80/tcp }; local ports_int: set[port] = { 80/tcp, 81/tcp, 443/tcp }; if (c$id$resp_h in ignore_host_resp) return; if (c$id$orig_h in ignore_host_orig) return; if (((c$id$resp_h in internal_space) && (c$id$resp_p in ports_int)) || ((c$id$resp_h in public_space) && (c$id$resp_p in ports_ext))) { if (c$http?$username && c$http?$status_code) { local auth_success : bool = T; if (c$http$status_code == 401) { auth_success = F; add c$http$tags[HTTP_AUTH_ERROR]; } else if (c$http$status_code < 400) { auth_success = T; add c$http$tags[HTTP_AUTH_SUCCESS]; } if(!auth_success) { SumStats::observe("http.auth_errors.attacker", [$host=to_addr(c$http$cluster_client_ip)], []); if ( c?$conn ) SumStats::observe("http.auth_errors.victim", [$host=c$conn$id$resp_h], []); } } } } https://github.com/michalpurzynski/bro-gramming/blob/ae37c0d6bfc62e25a797426d6791cf340b045d17/auth_bruteforcing.bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170405/fdf0734a/attachment-0001.html From bill.de.ping at gmail.com Wed Apr 5 05:40:37 2017 From: bill.de.ping at gmail.com (william de ping) Date: Wed, 5 Apr 2017 15:40:37 +0300 Subject: [Bro] minimalistic bro setup In-Reply-To: References: Message-ID: hi any ideas on how to turn off unwanted plugins\analyzers ? thanks On Tue, Apr 4, 2017 at 1:07 PM, william de ping wrote: > Hi all, > > I would like to make bro real thin by not loading all unnecessary > plugins\analyzers. > > I have tweaked init-bare and init-default scripts, yet when I see the > loaded-scripts, I see that many plugins are loaded. > > How can I turn off plugins effectively ? > when I edit base/bif/plugins/__load__.bro to not load ,say, FTP, I get > many errors that some FTP fields are not recognized and preventing the > cluster from running. > > I basically need only UDP and DNS events and have no need for the moment > for other down stream analyzers\plugins. > > Thanks in advance > B > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170405/8b5c9f87/attachment.html From fatema.bannatwala at gmail.com Wed Apr 5 05:59:01 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Wed, 5 Apr 2017 08:59:01 -0400 Subject: [Bro] auth_bruteforcing.bro error In-Reply-To: References: Message-ID: cluster_client_ip is the user defined field, http record doesn't have any field name "cluster_client_ip". I think what you want is c$http$id$orig_h , if that's what the purpose of cluster_client_ip is. Also your host is "string" type, you can change it to "addr" type: Might wanna try something like: type Info: record { * host: addr &log &optional;* }; SumStats::observe("http.auth_errors.attacker", [$host=c$http$id$orig_h], []); Also, not sure how this part is working(As c doesn't have "conn" field as well.): if ( c?$conn ) SumStats::observe("http.auth_errors.victim", [$host=c$conn$id$resp_h], []); On Wed, Apr 5, 2017 at 8:23 AM, ps sunu wrote: > > > I am using below code while running this i am getting below error from > below area > > > *if(!auth_success) {* > * SumStats::observe("http.auth_errors.attacker",* > * > [$host=to_addr(c$http$cluster_client_ip)],* > * []);* > * if ( c?$conn )* > > error > > > *field value missing [AuthBruteforcing::c$http$cluster_client_ip]* > code > > @load base/frameworks/notice > @load base/frameworks/sumstats > @load base/protocols/http > > module AuthBruteforcing; > > export { > redef enum Notice::Type += { > ## Indicates that a host performing HTTP requests leading to > ## excessive HTTP auth errors was detected. > HTTP_AuthBruteforcing_Attacker, > ## Indicates that a host was seen to respond excessive HTTP > ## auth errors. This is tracked by IP address as opposed to > ## hostname. > HTTP_AuthBruteforcing_Victim, > }; > > # Let's tag the http item > redef enum HTTP::Tags += { > ## HTTP status code 401, describing a HTTP auth error > HTTP_AUTH_ERROR, > ## HTTP describing a successful HTTP auth > HTTP_AUTH_SUCCESS, > }; > > redef enum Log::ID += { LOG }; > > type Info: record { > ts: time &log; > uid: string &log; > id: conn_id &log &optional; > cluster_client_ip: string &log &optional; > status_code: count &log &optional; > host: string &log &optional; > uri: string &log &optional; > username: string &log &optional; > auth_success: bool &log &optional; > }; > > global log_auth: event(rec: Info); > > ## Defines the threshold that determines if a auth bruteforcing attack > ## is ongoing based on the number of requests that appear to be > ## attacks. > const auth_errors_threshold: double = 50.0 &redef; > > ## Interval at which to watch for the > ## :bro:id:`AuthBruteforcing::auth_errors_requests_threshold` > variable to be crossed. > ## At the end of each interval the counter is reset. > const auth_errors_interval = 5min &redef; > > ## Interval at which to watch for the > ## :bro:id:`AuthBruteforcing::excessive_auth_errors_threshold` > variable to be > ## crossed. At the end of each interval the counter is reset. > const excessive_auth_errors_interval = 1min &redef; > > const internal_space: subnet = 10.0.0.0/8 &redef; > const public_space: subnet = 63.245.208.0/20 &redef; > > const ignore_host_resp: set[addr] = { } &redef; > const ignore_host_orig: set[addr] = { } &redef; > } > > event bro_init() &priority=3 > { > # Create auth_bruteforcing.log > Log::create_stream(AuthBruteforcing::LOG, [$columns=Info, > $ev=log_auth]); > > # HTTP auth errors for requests FROM the same host > local r1: SumStats::Reducer = [$stream="http.auth_errors.attacker", > $apply=set(SumStats::SUM)]; > SumStats::create([$name="auth-http-errors-attackers", > $epoch=auth_errors_interval, > $reducers=set(r1), > $threshold_val(key: SumStats::Key, result: > SumStats::Result) = { > return result["http.auth_errors.attacker"]$sum; > }, > $threshold=auth_errors_threshold, > $threshold_crossed(key: SumStats::Key, result: > SumStats::Result) = { > NOTICE([$note=HTTP_AuthBruteforcing_Attacker, > $msg=fmt("HTTP auth bruteforcing from > attacker %s", key$host), > $sub=fmt("%.0f auth failed in %s", > result["http.auth_errors.attacker"]$sum, auth_errors_interval), > $src=key$host, > $n=to_count(fmt("%.0f", > result["http.auth_errors.attacker"]$sum)) > ]); > }]); > > # HTTP errors for requests TO the same host > local r2: SumStats::Reducer = [$stream="http.auth_errors.victim", > $apply=set(SumStats::SUM)]; > SumStats::create([$name="auth-http-errors-victims", > $epoch=auth_errors_interval, > $reducers=set(r2), > $threshold_val(key: SumStats::Key, result: > SumStats::Result) = { > return result["http.auth_errors.victim"]$sum; > }, > $threshold=auth_errors_threshold, > $threshold_crossed(key: SumStats::Key, result: > SumStats::Result) = { > NOTICE([$note=HTTP_AuthBruteforcing_Victim, > $msg=fmt("HTTP auth bruteforcing to > victim %s", key$host), > $sub=fmt("%.0f auth failed in %s", > result["http.auth_errors.victim"]$sum, auth_errors_interval), > $src=key$host, > $n=to_count(fmt("%.0f", > result["http.auth_errors.victim"]$sum)) > ]); > }]); > } > > # Make sure we have all the http info before looking for auth errors > event http_message_done(c: connection, is_orig: bool, stat: > http_message_stat) > { > # only conns we want > local ports_ext: set[port] = { 80/tcp }; > local ports_int: set[port] = { 80/tcp, 81/tcp, 443/tcp }; > > if (c$id$resp_h in ignore_host_resp) > return; > if (c$id$orig_h in ignore_host_orig) > return; > > if (((c$id$resp_h in internal_space) && (c$id$resp_p in ports_int)) || > ((c$id$resp_h in public_space) && (c$id$resp_p in ports_ext))) { > > if (c$http?$username && c$http?$status_code) { > local auth_success : bool = T; > if (c$http$status_code == 401) { > auth_success = F; > add c$http$tags[HTTP_AUTH_ERROR]; > } > else if (c$http$status_code < 400) { > auth_success = T; > add c$http$tags[HTTP_AUTH_SUCCESS]; > } > if(!auth_success) { > SumStats::observe("http.auth_errors.attacker", > [$host=to_addr(c$http$cluster_ > client_ip)], > []); > if ( c?$conn ) > SumStats::observe("http.auth_errors.victim", > [$host=c$conn$id$resp_h], > []); > } > } > } > } > > > > https://github.com/michalpurzynski/bro-gramming/blob/ > ae37c0d6bfc62e25a797426d6791cf340b045d17/auth_bruteforcing.bro > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170405/ec703a28/attachment-0001.html From philosnef at gmail.com Wed Apr 5 06:02:51 2017 From: philosnef at gmail.com (erik clark) Date: Wed, 5 Apr 2017 09:02:51 -0400 Subject: [Bro] Yara integration with Bro 2.5 Message-ID: Er, doesnt this come with massive overhead? Also, file inspection rules are non-trivial. Given the number of files that bro processes, it seems that on anything other than a very tiny link this would cause giant problems... -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170405/e2160eaa/attachment.html From pssunu6 at gmail.com Wed Apr 5 06:49:34 2017 From: pssunu6 at gmail.com (ps sunu) Date: Wed, 5 Apr 2017 19:19:34 +0530 Subject: [Bro] auth_bruteforcing.bro error In-Reply-To: References: Message-ID: i cleared using below code if( c$http?$cluster_client_ip ) On Wed, Apr 5, 2017 at 6:29 PM, fatema bannatwala < fatema.bannatwala at gmail.com> wrote: > cluster_client_ip is the user defined field, http record doesn't have any > field name "cluster_client_ip". > I think what you want is c$http$id$orig_h , if that's what the purpose of > cluster_client_ip is. > Also your host is "string" type, you can change it to "addr" type: > > Might wanna try something like: > > type Info: record { > > * host: addr &log &optional;* > > }; > > SumStats::observe("http.auth_errors.attacker", > [$host=c$http$id$orig_h], > []); > > Also, not sure how this part is working(As c doesn't have "conn" field as > well.): > > if ( c?$conn ) > SumStats::observe("http.auth_errors.victim", > [$host=c$conn$id$resp_h], > []); > > > > On Wed, Apr 5, 2017 at 8:23 AM, ps sunu wrote: > >> >> >> I am using below code while running this i am getting below error from >> below area >> >> >> *if(!auth_success) {* >> * SumStats::observe("http.auth_errors.attacker",* >> * >> [$host=to_addr(c$http$cluster_client_ip)],* >> * []);* >> * if ( c?$conn )* >> >> error >> >> >> *field value missing [AuthBruteforcing::c$http$cluster_client_ip]* >> code >> >> @load base/frameworks/notice >> @load base/frameworks/sumstats >> @load base/protocols/http >> >> module AuthBruteforcing; >> >> export { >> redef enum Notice::Type += { >> ## Indicates that a host performing HTTP requests leading to >> ## excessive HTTP auth errors was detected. >> HTTP_AuthBruteforcing_Attacker, >> ## Indicates that a host was seen to respond excessive HTTP >> ## auth errors. This is tracked by IP address as opposed to >> ## hostname. >> HTTP_AuthBruteforcing_Victim, >> }; >> >> # Let's tag the http item >> redef enum HTTP::Tags += { >> ## HTTP status code 401, describing a HTTP auth error >> HTTP_AUTH_ERROR, >> ## HTTP describing a successful HTTP auth >> HTTP_AUTH_SUCCESS, >> }; >> >> redef enum Log::ID += { LOG }; >> >> type Info: record { >> ts: time &log; >> uid: string &log; >> id: conn_id &log &optional; >> cluster_client_ip: string &log &optional; >> status_code: count &log &optional; >> host: string &log &optional; >> uri: string &log &optional; >> username: string &log &optional; >> auth_success: bool &log &optional; >> }; >> >> global log_auth: event(rec: Info); >> >> ## Defines the threshold that determines if a auth bruteforcing attack >> ## is ongoing based on the number of requests that appear to be >> ## attacks. >> const auth_errors_threshold: double = 50.0 &redef; >> >> ## Interval at which to watch for the >> ## :bro:id:`AuthBruteforcing::auth_errors_requests_threshold` >> variable to be crossed. >> ## At the end of each interval the counter is reset. >> const auth_errors_interval = 5min &redef; >> >> ## Interval at which to watch for the >> ## :bro:id:`AuthBruteforcing::excessive_auth_errors_threshold` >> variable to be >> ## crossed. At the end of each interval the counter is reset. >> const excessive_auth_errors_interval = 1min &redef; >> >> const internal_space: subnet = 10.0.0.0/8 &redef; >> const public_space: subnet = 63.245.208.0/20 &redef; >> >> const ignore_host_resp: set[addr] = { } &redef; >> const ignore_host_orig: set[addr] = { } &redef; >> } >> >> event bro_init() &priority=3 >> { >> # Create auth_bruteforcing.log >> Log::create_stream(AuthBruteforcing::LOG, [$columns=Info, >> $ev=log_auth]); >> >> # HTTP auth errors for requests FROM the same host >> local r1: SumStats::Reducer = [$stream="http.auth_errors.attacker", >> $apply=set(SumStats::SUM)]; >> SumStats::create([$name="auth-http-errors-attackers", >> $epoch=auth_errors_interval, >> $reducers=set(r1), >> $threshold_val(key: SumStats::Key, result: >> SumStats::Result) = { >> return result["http.auth_errors.attacker"]$sum; >> }, >> $threshold=auth_errors_threshold, >> $threshold_crossed(key: SumStats::Key, result: >> SumStats::Result) = { >> NOTICE([$note=HTTP_AuthBruteforcing_Attacker, >> $msg=fmt("HTTP auth bruteforcing from >> attacker %s", key$host), >> $sub=fmt("%.0f auth failed in %s", >> result["http.auth_errors.attacker"]$sum, auth_errors_interval), >> $src=key$host, >> $n=to_count(fmt("%.0f", >> result["http.auth_errors.attacker"]$sum)) >> ]); >> }]); >> >> # HTTP errors for requests TO the same host >> local r2: SumStats::Reducer = [$stream="http.auth_errors.victim", >> $apply=set(SumStats::SUM)]; >> SumStats::create([$name="auth-http-errors-victims", >> $epoch=auth_errors_interval, >> $reducers=set(r2), >> $threshold_val(key: SumStats::Key, result: >> SumStats::Result) = { >> return result["http.auth_errors.victim"]$sum; >> }, >> $threshold=auth_errors_threshold, >> $threshold_crossed(key: SumStats::Key, result: >> SumStats::Result) = { >> NOTICE([$note=HTTP_AuthBruteforcing_Victim, >> $msg=fmt("HTTP auth bruteforcing to >> victim %s", key$host), >> $sub=fmt("%.0f auth failed in %s", >> result["http.auth_errors.victim"]$sum, auth_errors_interval), >> $src=key$host, >> $n=to_count(fmt("%.0f", >> result["http.auth_errors.victim"]$sum)) >> ]); >> }]); >> } >> >> # Make sure we have all the http info before looking for auth errors >> event http_message_done(c: connection, is_orig: bool, stat: >> http_message_stat) >> { >> # only conns we want >> local ports_ext: set[port] = { 80/tcp }; >> local ports_int: set[port] = { 80/tcp, 81/tcp, 443/tcp }; >> >> if (c$id$resp_h in ignore_host_resp) >> return; >> if (c$id$orig_h in ignore_host_orig) >> return; >> >> if (((c$id$resp_h in internal_space) && (c$id$resp_p in ports_int)) >> || ((c$id$resp_h in public_space) && (c$id$resp_p in ports_ext))) { >> >> if (c$http?$username && c$http?$status_code) { >> local auth_success : bool = T; >> if (c$http$status_code == 401) { >> auth_success = F; >> add c$http$tags[HTTP_AUTH_ERROR]; >> } >> else if (c$http$status_code < 400) { >> auth_success = T; >> add c$http$tags[HTTP_AUTH_SUCCESS]; >> } >> if(!auth_success) { >> SumStats::observe("http.auth_errors.attacker", >> [$host=to_addr(c$http$cluster_ >> client_ip)], >> []); >> if ( c?$conn ) >> SumStats::observe("http.auth_errors.victim", >> [$host=c$conn$id$resp_h], >> []); >> } >> } >> } >> } >> >> >> >> https://github.com/michalpurzynski/bro-gramming/blob/ae37c0d >> 6bfc62e25a797426d6791cf340b045d17/auth_bruteforcing.bro >> >> >> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170405/63f08938/attachment-0001.html From pssunu6 at gmail.com Wed Apr 5 06:50:51 2017 From: pssunu6 at gmail.com (ps sunu) Date: Wed, 5 Apr 2017 19:20:51 +0530 Subject: [Bro] auth_bruteforcing.bro error In-Reply-To: References: Message-ID: thanks On Wed, Apr 5, 2017 at 7:19 PM, ps sunu wrote: > i cleared using below code > > if( c$http?$cluster_client_ip ) > > On Wed, Apr 5, 2017 at 6:29 PM, fatema bannatwala < > fatema.bannatwala at gmail.com> wrote: > >> cluster_client_ip is the user defined field, http record doesn't have any >> field name "cluster_client_ip". >> I think what you want is c$http$id$orig_h , if that's what the purpose >> of cluster_client_ip is. >> Also your host is "string" type, you can change it to "addr" type: >> >> Might wanna try something like: >> >> type Info: record { >> >> * host: addr &log &optional;* >> >> }; >> >> SumStats::observe("http.auth_errors.attacker", >> [$host=c$http$id$orig_h], >> []); >> >> Also, not sure how this part is working(As c doesn't have "conn" field as >> well.): >> >> if ( c?$conn ) >> SumStats::observe("http.auth_errors.victim", >> [$host=c$conn$id$resp_h], >> []); >> >> >> >> On Wed, Apr 5, 2017 at 8:23 AM, ps sunu wrote: >> >>> >>> >>> I am using below code while running this i am getting below error from >>> below area >>> >>> >>> *if(!auth_success) {* >>> * SumStats::observe("http.auth_errors.attacker",* >>> * >>> [$host=to_addr(c$http$cluster_client_ip)],* >>> * []);* >>> * if ( c?$conn )* >>> >>> error >>> >>> >>> *field value missing [AuthBruteforcing::c$http$cluster_client_ip]* >>> code >>> >>> @load base/frameworks/notice >>> @load base/frameworks/sumstats >>> @load base/protocols/http >>> >>> module AuthBruteforcing; >>> >>> export { >>> redef enum Notice::Type += { >>> ## Indicates that a host performing HTTP requests leading to >>> ## excessive HTTP auth errors was detected. >>> HTTP_AuthBruteforcing_Attacker, >>> ## Indicates that a host was seen to respond excessive HTTP >>> ## auth errors. This is tracked by IP address as opposed to >>> ## hostname. >>> HTTP_AuthBruteforcing_Victim, >>> }; >>> >>> # Let's tag the http item >>> redef enum HTTP::Tags += { >>> ## HTTP status code 401, describing a HTTP auth error >>> HTTP_AUTH_ERROR, >>> ## HTTP describing a successful HTTP auth >>> HTTP_AUTH_SUCCESS, >>> }; >>> >>> redef enum Log::ID += { LOG }; >>> >>> type Info: record { >>> ts: time &log; >>> uid: string &log; >>> id: conn_id &log &optional; >>> cluster_client_ip: string &log &optional; >>> status_code: count &log &optional; >>> host: string &log &optional; >>> uri: string &log &optional; >>> username: string &log &optional; >>> auth_success: bool &log &optional; >>> }; >>> >>> global log_auth: event(rec: Info); >>> >>> ## Defines the threshold that determines if a auth bruteforcing >>> attack >>> ## is ongoing based on the number of requests that appear to be >>> ## attacks. >>> const auth_errors_threshold: double = 50.0 &redef; >>> >>> ## Interval at which to watch for the >>> ## :bro:id:`AuthBruteforcing::auth_errors_requests_threshold` >>> variable to be crossed. >>> ## At the end of each interval the counter is reset. >>> const auth_errors_interval = 5min &redef; >>> >>> ## Interval at which to watch for the >>> ## :bro:id:`AuthBruteforcing::excessive_auth_errors_threshold` >>> variable to be >>> ## crossed. At the end of each interval the counter is reset. >>> const excessive_auth_errors_interval = 1min &redef; >>> >>> const internal_space: subnet = 10.0.0.0/8 &redef; >>> const public_space: subnet = 63.245.208.0/20 &redef; >>> >>> const ignore_host_resp: set[addr] = { } &redef; >>> const ignore_host_orig: set[addr] = { } &redef; >>> } >>> >>> event bro_init() &priority=3 >>> { >>> # Create auth_bruteforcing.log >>> Log::create_stream(AuthBruteforcing::LOG, [$columns=Info, >>> $ev=log_auth]); >>> >>> # HTTP auth errors for requests FROM the same host >>> local r1: SumStats::Reducer = [$stream="http.auth_errors.attacker", >>> $apply=set(SumStats::SUM)]; >>> SumStats::create([$name="auth-http-errors-attackers", >>> $epoch=auth_errors_interval, >>> $reducers=set(r1), >>> $threshold_val(key: SumStats::Key, result: >>> SumStats::Result) = { >>> return result["http.auth_errors.attac >>> ker"]$sum; >>> }, >>> $threshold=auth_errors_threshold, >>> $threshold_crossed(key: SumStats::Key, result: >>> SumStats::Result) = { >>> NOTICE([$note=HTTP_AuthBruteforcing_Attacker, >>> $msg=fmt("HTTP auth bruteforcing from >>> attacker %s", key$host), >>> $sub=fmt("%.0f auth failed in %s", >>> result["http.auth_errors.attacker"]$sum, auth_errors_interval), >>> $src=key$host, >>> $n=to_count(fmt("%.0f", >>> result["http.auth_errors.attacker"]$sum)) >>> ]); >>> }]); >>> >>> # HTTP errors for requests TO the same host >>> local r2: SumStats::Reducer = [$stream="http.auth_errors.victim", >>> $apply=set(SumStats::SUM)]; >>> SumStats::create([$name="auth-http-errors-victims", >>> $epoch=auth_errors_interval, >>> $reducers=set(r2), >>> $threshold_val(key: SumStats::Key, result: >>> SumStats::Result) = { >>> return result["http.auth_errors.victim"]$sum; >>> }, >>> $threshold=auth_errors_threshold, >>> $threshold_crossed(key: SumStats::Key, result: >>> SumStats::Result) = { >>> NOTICE([$note=HTTP_AuthBruteforcing_Victim, >>> $msg=fmt("HTTP auth bruteforcing to >>> victim %s", key$host), >>> $sub=fmt("%.0f auth failed in %s", >>> result["http.auth_errors.victim"]$sum, auth_errors_interval), >>> $src=key$host, >>> $n=to_count(fmt("%.0f", >>> result["http.auth_errors.victim"]$sum)) >>> ]); >>> }]); >>> } >>> >>> # Make sure we have all the http info before looking for auth errors >>> event http_message_done(c: connection, is_orig: bool, stat: >>> http_message_stat) >>> { >>> # only conns we want >>> local ports_ext: set[port] = { 80/tcp }; >>> local ports_int: set[port] = { 80/tcp, 81/tcp, 443/tcp }; >>> >>> if (c$id$resp_h in ignore_host_resp) >>> return; >>> if (c$id$orig_h in ignore_host_orig) >>> return; >>> >>> if (((c$id$resp_h in internal_space) && (c$id$resp_p in ports_int)) >>> || ((c$id$resp_h in public_space) && (c$id$resp_p in ports_ext))) { >>> >>> if (c$http?$username && c$http?$status_code) { >>> local auth_success : bool = T; >>> if (c$http$status_code == 401) { >>> auth_success = F; >>> add c$http$tags[HTTP_AUTH_ERROR]; >>> } >>> else if (c$http$status_code < 400) { >>> auth_success = T; >>> add c$http$tags[HTTP_AUTH_SUCCESS]; >>> } >>> if(!auth_success) { >>> SumStats::observe("http.auth_errors.attacker", >>> [$host=to_addr(c$http$cluster_ >>> client_ip)], >>> []); >>> if ( c?$conn ) >>> SumStats::observe("http.auth_errors.victim", >>> [$host=c$conn$id$resp_h], >>> []); >>> } >>> } >>> } >>> } >>> >>> >>> >>> https://github.com/michalpurzynski/bro-gramming/blob/ae37c0d >>> 6bfc62e25a797426d6791cf340b045d17/auth_bruteforcing.bro >>> >>> >>> >>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170405/c3d3fb18/attachment.html From fatema.bannatwala at gmail.com Wed Apr 5 07:32:21 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Wed, 5 Apr 2017 10:32:21 -0400 Subject: [Bro] auth_bruteforcing.bro error In-Reply-To: References: Message-ID: >if( c$http?$cluster_client_ip ) Though, I wonder if this condition is ever going to result in true... On Wed, Apr 5, 2017 at 9:49 AM, ps sunu wrote: > i cleared using below code > > if( c$http?$cluster_client_ip ) > > On Wed, Apr 5, 2017 at 6:29 PM, fatema bannatwala < > fatema.bannatwala at gmail.com> wrote: > >> cluster_client_ip is the user defined field, http record doesn't have any >> field name "cluster_client_ip". >> I think what you want is c$http$id$orig_h , if that's what the purpose >> of cluster_client_ip is. >> Also your host is "string" type, you can change it to "addr" type: >> >> Might wanna try something like: >> >> type Info: record { >> >> * host: addr &log &optional;* >> >> }; >> >> SumStats::observe("http.auth_errors.attacker", >> [$host=c$http$id$orig_h], >> []); >> >> Also, not sure how this part is working(As c doesn't have "conn" field as >> well.): >> >> if ( c?$conn ) >> SumStats::observe("http.auth_errors.victim", >> [$host=c$conn$id$resp_h], >> []); >> >> >> >> On Wed, Apr 5, 2017 at 8:23 AM, ps sunu wrote: >> >>> >>> >>> I am using below code while running this i am getting below error from >>> below area >>> >>> >>> *if(!auth_success) {* >>> * SumStats::observe("http.auth_errors.attacker",* >>> * >>> [$host=to_addr(c$http$cluster_client_ip)],* >>> * []);* >>> * if ( c?$conn )* >>> >>> error >>> >>> >>> *field value missing [AuthBruteforcing::c$http$cluster_client_ip]* >>> code >>> >>> @load base/frameworks/notice >>> @load base/frameworks/sumstats >>> @load base/protocols/http >>> >>> module AuthBruteforcing; >>> >>> export { >>> redef enum Notice::Type += { >>> ## Indicates that a host performing HTTP requests leading to >>> ## excessive HTTP auth errors was detected. >>> HTTP_AuthBruteforcing_Attacker, >>> ## Indicates that a host was seen to respond excessive HTTP >>> ## auth errors. This is tracked by IP address as opposed to >>> ## hostname. >>> HTTP_AuthBruteforcing_Victim, >>> }; >>> >>> # Let's tag the http item >>> redef enum HTTP::Tags += { >>> ## HTTP status code 401, describing a HTTP auth error >>> HTTP_AUTH_ERROR, >>> ## HTTP describing a successful HTTP auth >>> HTTP_AUTH_SUCCESS, >>> }; >>> >>> redef enum Log::ID += { LOG }; >>> >>> type Info: record { >>> ts: time &log; >>> uid: string &log; >>> id: conn_id &log &optional; >>> cluster_client_ip: string &log &optional; >>> status_code: count &log &optional; >>> host: string &log &optional; >>> uri: string &log &optional; >>> username: string &log &optional; >>> auth_success: bool &log &optional; >>> }; >>> >>> global log_auth: event(rec: Info); >>> >>> ## Defines the threshold that determines if a auth bruteforcing >>> attack >>> ## is ongoing based on the number of requests that appear to be >>> ## attacks. >>> const auth_errors_threshold: double = 50.0 &redef; >>> >>> ## Interval at which to watch for the >>> ## :bro:id:`AuthBruteforcing::auth_errors_requests_threshold` >>> variable to be crossed. >>> ## At the end of each interval the counter is reset. >>> const auth_errors_interval = 5min &redef; >>> >>> ## Interval at which to watch for the >>> ## :bro:id:`AuthBruteforcing::excessive_auth_errors_threshold` >>> variable to be >>> ## crossed. At the end of each interval the counter is reset. >>> const excessive_auth_errors_interval = 1min &redef; >>> >>> const internal_space: subnet = 10.0.0.0/8 &redef; >>> const public_space: subnet = 63.245.208.0/20 &redef; >>> >>> const ignore_host_resp: set[addr] = { } &redef; >>> const ignore_host_orig: set[addr] = { } &redef; >>> } >>> >>> event bro_init() &priority=3 >>> { >>> # Create auth_bruteforcing.log >>> Log::create_stream(AuthBruteforcing::LOG, [$columns=Info, >>> $ev=log_auth]); >>> >>> # HTTP auth errors for requests FROM the same host >>> local r1: SumStats::Reducer = [$stream="http.auth_errors.attacker", >>> $apply=set(SumStats::SUM)]; >>> SumStats::create([$name="auth-http-errors-attackers", >>> $epoch=auth_errors_interval, >>> $reducers=set(r1), >>> $threshold_val(key: SumStats::Key, result: >>> SumStats::Result) = { >>> return result["http.auth_errors.attac >>> ker"]$sum; >>> }, >>> $threshold=auth_errors_threshold, >>> $threshold_crossed(key: SumStats::Key, result: >>> SumStats::Result) = { >>> NOTICE([$note=HTTP_AuthBruteforcing_Attacker, >>> $msg=fmt("HTTP auth bruteforcing from >>> attacker %s", key$host), >>> $sub=fmt("%.0f auth failed in %s", >>> result["http.auth_errors.attacker"]$sum, auth_errors_interval), >>> $src=key$host, >>> $n=to_count(fmt("%.0f", >>> result["http.auth_errors.attacker"]$sum)) >>> ]); >>> }]); >>> >>> # HTTP errors for requests TO the same host >>> local r2: SumStats::Reducer = [$stream="http.auth_errors.victim", >>> $apply=set(SumStats::SUM)]; >>> SumStats::create([$name="auth-http-errors-victims", >>> $epoch=auth_errors_interval, >>> $reducers=set(r2), >>> $threshold_val(key: SumStats::Key, result: >>> SumStats::Result) = { >>> return result["http.auth_errors.victim"]$sum; >>> }, >>> $threshold=auth_errors_threshold, >>> $threshold_crossed(key: SumStats::Key, result: >>> SumStats::Result) = { >>> NOTICE([$note=HTTP_AuthBruteforcing_Victim, >>> $msg=fmt("HTTP auth bruteforcing to >>> victim %s", key$host), >>> $sub=fmt("%.0f auth failed in %s", >>> result["http.auth_errors.victim"]$sum, auth_errors_interval), >>> $src=key$host, >>> $n=to_count(fmt("%.0f", >>> result["http.auth_errors.victim"]$sum)) >>> ]); >>> }]); >>> } >>> >>> # Make sure we have all the http info before looking for auth errors >>> event http_message_done(c: connection, is_orig: bool, stat: >>> http_message_stat) >>> { >>> # only conns we want >>> local ports_ext: set[port] = { 80/tcp }; >>> local ports_int: set[port] = { 80/tcp, 81/tcp, 443/tcp }; >>> >>> if (c$id$resp_h in ignore_host_resp) >>> return; >>> if (c$id$orig_h in ignore_host_orig) >>> return; >>> >>> if (((c$id$resp_h in internal_space) && (c$id$resp_p in ports_int)) >>> || ((c$id$resp_h in public_space) && (c$id$resp_p in ports_ext))) { >>> >>> if (c$http?$username && c$http?$status_code) { >>> local auth_success : bool = T; >>> if (c$http$status_code == 401) { >>> auth_success = F; >>> add c$http$tags[HTTP_AUTH_ERROR]; >>> } >>> else if (c$http$status_code < 400) { >>> auth_success = T; >>> add c$http$tags[HTTP_AUTH_SUCCESS]; >>> } >>> if(!auth_success) { >>> SumStats::observe("http.auth_errors.attacker", >>> [$host=to_addr(c$http$cluster_ >>> client_ip)], >>> []); >>> if ( c?$conn ) >>> SumStats::observe("http.auth_errors.victim", >>> [$host=c$conn$id$resp_h], >>> []); >>> } >>> } >>> } >>> } >>> >>> >>> >>> https://github.com/michalpurzynski/bro-gramming/blob/ae37c0d >>> 6bfc62e25a797426d6791cf340b045d17/auth_bruteforcing.bro >>> >>> >>> >>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170405/ef5a5772/attachment-0001.html From johanna at icir.org Wed Apr 5 09:21:06 2017 From: johanna at icir.org (Johanna Amann) Date: Wed, 5 Apr 2017 09:21:06 -0700 Subject: [Bro] minimalistic bro setup In-Reply-To: References: Message-ID: <20170405162103.27agsuge6b4wl6vm@wifi240.sys.ICSI.Berkeley.EDU> You are probably looking for bare mode, which you can use by starting Bro with the "-b" option. In bare mode, Bro only loads init-bare.bro, and does not load init-default; thus basically no analyzers are activated. Johanna On Wed, Apr 05, 2017 at 03:40:37PM +0300, william de ping wrote: > hi > any ideas on how to turn off unwanted plugins\analyzers ? > > thanks > > On Tue, Apr 4, 2017 at 1:07 PM, william de ping > wrote: > > > Hi all, > > > > I would like to make bro real thin by not loading all unnecessary > > plugins\analyzers. > > > > I have tweaked init-bare and init-default scripts, yet when I see the > > loaded-scripts, I see that many plugins are loaded. > > > > How can I turn off plugins effectively ? > > when I edit base/bif/plugins/__load__.bro to not load ,say, FTP, I get > > many errors that some FTP fields are not recognized and preventing the > > cluster from running. > > > > I basically need only UDP and DNS events and have no need for the moment > > for other down stream analyzers\plugins. > > > > Thanks in advance > > B > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Wed Apr 5 09:28:46 2017 From: johanna at icir.org (Johanna Amann) Date: Wed, 5 Apr 2017 09:28:46 -0700 Subject: [Bro] TCP Conn Log In-Reply-To: References: Message-ID: <20170405162846.rhlxe63lul4i3lfq@wifi240.sys.ICSI.Berkeley.EDU> Hi Mike, I am currently not aware of any way to accomplish this without modifications to the core. You can change the timeout that Bro uses for TCP connections (the time after which its expires a connection, if it does not see any packets) by changing tcp_inactivity_timeout; depending on your specific application, maybe that might be good enough. Johanna On Mon, Apr 03, 2017 at 02:49:21PM +0200, mike anastasakis wrote: > Hello, > > I am using Bro for a project and I have a question regarding it's > capabilities. > Currently when I have a long TCP connection that includes frequent TCP Keep > Alive messages, bro is reassembling the whole network trace into one > connection and presents it in conn.log with a big duration value. Is it > possible to make bro split up TCP connections into smaller fragments based > on a interval I set up or at least whenever a TCP Keep alive handshake > takes place? > > > Regards, > Mike > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Wed Apr 5 09:30:51 2017 From: johanna at icir.org (Johanna Amann) Date: Wed, 5 Apr 2017 09:30:51 -0700 Subject: [Bro] NetControl configuration In-Reply-To: References: <9d900f340345412d8c545f2ced222966@moxde9.na.bayer.cnb> Message-ID: <20170405163051.x2f4oqrb2wik22wc@wifi240.sys.ICSI.Berkeley.EDU> Hi, The script excerpt is not quite long enough to see what exactly is going on here (it does, for example, not show where conn_id is coming from and how you defined it). Could you perhaps just post the complete script in its current state? Johanna On Thu, Mar 30, 2017 at 02:32:51PM +0000, Andrew Dellana wrote: > Got around to adding net control to all the scripts, and now they are failing. The script is FoxIT's ransomware script. Any idea how I can get this to work? > > > event NetControl::init() > { > NetControl::drop_connection (conn_id, 0, "Cyrpto Blocked") > } > > > hook Notice::policy(n: Notice::Info) > { > if fox_entropy=T Then > add n$actions[Notice::ACTION_DROP] > add n$actions[Notice::ACTION_EMAIL]; > } > > > > > error in /opt/bro/share/bro/base/init-bare.bro, lines 123-127 and /opt/bro/share/bro/base/misc/CryptoRansomCheck.bro, line 127: type clash (conn_id and conn_id) > error in /opt/bro/share/bro/base/misc/CryptoRansomCheck.bro, line 127 and /opt/bro/share/bro/base/init-bare.bro, lines 123-127: type mismatch (conn_id and conn_id) > error in /opt/bro/share/bro/base/misc/CryptoRansomCheck.bro, line 127: argument type mismatch in function call (NetControl::drop_connection(conn_id, 0, Cyrpto Blocked)) > error in /opt/bro/share/bro/base/misc/CryptoRansomCheck.bro, line 128: syntax error, at or near "}" > > > Freundliche Gr??e / Best regards, > > Andrew Dellana > Intern > ________________________ > > > -----Original Message----- > From: Azoff, Justin S [mailto:jazoff at illinois.edu] > Sent: Thursday, March 16, 2017 11:08 AM > To: Andrew Dellana > Cc: bro at bro.org > Subject: Re: [Bro] NetControl configuration > > > > On Mar 16, 2017, at 11:04 AM, Andrew Dellana wrote: > > > > Yes, I do want to make the NetControl actions based on what is alerted in Notices. Can all the helpers be stored in one file and only call the helper that is needed? > > Yep, you can do exactly that. > > -- > - Justin Azoff > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From johanna at icir.org Wed Apr 5 09:34:08 2017 From: johanna at icir.org (Johanna Amann) Date: Wed, 5 Apr 2017 09:34:08 -0700 Subject: [Bro] &log cert_chain attribute (vector of Files::info) in ssl.log file In-Reply-To: References: Message-ID: <20170405163407.cmqsoo4pssakxp7w@wifi240.sys.ICSI.Berkeley.EDU> Hi, yes, you are right, cert_chain can currently not be logged. The logging framework is limited to fields that can be represented in ASCII columns; hence only vectors of base-types can be logged. Files::info is not a base type :). What exactly do you want to log? The hashes? In this case, the way to do this is to add an event handler that takes the information in cert_chain, transforms it into a format that can be logged, and writes it into another field. For example, if you want to log the certificate hashes, you would go through the cert_chain, extract all hashes, and then write them to a field of type vector of string, which can be logged. I hope this helps :) Johanna On Tue, Mar 28, 2017 at 01:33:51PM -0400, Robert Harrelson wrote: > &log cert_chain attribute (vector of Files::info) in ssl.log file. > > I would like to list the server's chain of certificates in ssl.log (log of > handshake data) alongside each handshake. > > In ssl.log, the cert_chain attribute (certificate chain of the server) is > not being logged, and is of type *vector of **Files::info*. When I tried to > add "&log" attribute to cert_chain in files.bro, it gave an error that: > > ".... cert_chain is of type that cannot be logged." > > When I tried changing the type from *vector of Files::info* to *vector of > string*, it sprang up some different errors since cert_chain is referenced > as a *vector of Files::info* in other parts of files.bro script. > > Please tell me how I can log the cert_chain attribute in ssl.log file. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Wed Apr 5 09:36:15 2017 From: johanna at icir.org (Johanna Amann) Date: Wed, 5 Apr 2017 09:36:15 -0700 Subject: [Bro] dpdk In-Reply-To: References: Message-ID: <20170405163615.xfzottsicfjlghou@wifi240.sys.ICSI.Berkeley.EDU> I am not aware of anyone currently working on this. Since Bro supports plugins for iosources, this could be added by anyone as a plugin (which would even be installable using bro-pkg). Johanna On Mon, Mar 27, 2017 at 12:07:13PM -0400, erik clark wrote: > Any idea if this will be supported? I can not find any reference in the > past year indicating this one way or another. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Wed Apr 5 09:37:29 2017 From: johanna at icir.org (Johanna Amann) Date: Wed, 5 Apr 2017 09:37:29 -0700 Subject: [Bro] log rotation In-Reply-To: References: Message-ID: <20170405163729.o44epofzhvo5lir7@wifi240.sys.ICSI.Berkeley.EDU> Hi Asad, Bro currently does not support appending data to the same log file over several runs. The typical way to solve this is to have a script which generates a new directory for each run, automatically changes the working dorectory to it, and runs Bro from there. Afterwards you can concatenate the output files. I hope this helps, Johanna On Sun, Mar 26, 2017 at 06:18:03PM +0000, Ul Asad, Hafiz wrote: > Hi, > > I am analysing a large number of "pcap" files using, > > bro -r *.pcap my_bro.script > > The problem is that for each new pcap file, bro over-writes the previous *.log files if I don't change my working directory. Is there a way of controlling the rotation of log files? I know that "broctl" has this time base rotation, but is there any sort of rotation control when bro is run from command line? I can change the working directory, but I want to have all my results in a single a log file (files) so that it is easy to query them. > > Regards > Asad > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jazoff at illinois.edu Wed Apr 5 09:40:52 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 5 Apr 2017 16:40:52 +0000 Subject: [Bro] TCP Conn Log In-Reply-To: <20170405162846.rhlxe63lul4i3lfq@wifi240.sys.ICSI.Berkeley.EDU> References: <20170405162846.rhlxe63lul4i3lfq@wifi240.sys.ICSI.Berkeley.EDU> Message-ID: Oh! there is this script that may help: https://github.com/corelight/bro-long-connections -- - Justin Azoff > On Apr 5, 2017, at 12:28 PM, Johanna Amann wrote: > > Hi Mike, > > I am currently not aware of any way to accomplish this without > modifications to the core. You can change the timeout that Bro uses for > TCP connections (the time after which its expires a connection, if it does > not see any packets) by changing tcp_inactivity_timeout; depending on your > specific application, maybe that might be good enough. > > Johanna > > On Mon, Apr 03, 2017 at 02:49:21PM +0200, mike anastasakis wrote: >> Hello, >> >> I am using Bro for a project and I have a question regarding it's >> capabilities. >> Currently when I have a long TCP connection that includes frequent TCP Keep >> Alive messages, bro is reassembling the whole network trace into one >> connection and presents it in conn.log with a big duration value. Is it >> possible to make bro split up TCP connections into smaller fragments based >> on a interval I set up or at least whenever a TCP Keep alive handshake >> takes place? >> >> >> Regards, >> Mike > >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Wed Apr 5 09:42:06 2017 From: johanna at icir.org (Johanna Amann) Date: Wed, 5 Apr 2017 09:42:06 -0700 Subject: [Bro] Getting 'standard' Bro events into Python In-Reply-To: References: Message-ID: <20170405164206.25iv3wlagnkpohhm@wifi240.sys.ICSI.Berkeley.EDU> Hi Brian, you are right that Broker is the new communication library; please note that the API is not quite finished yet and that you will have to adjust your code when the next Bro version is release. Note that, for both broker and broccoli, you will not just be able to receive connection (or other) events; instead you will have to handle them in a bro event where you can re-throw them (...under a different name, to not cause issues with other scripts). If you subscribe to that new event using broker, you should be able to receive data. The best example for using broker to communicate with Bro, that currently exists, are probably the netcontrol adapters; an easy example is available at https://github.com/bro/bro-netcontrol/tree/master/test I hope this helps, Johanna On Fri, Mar 24, 2017 at 11:54:52AM -0600, Brian Wylie wrote: > Okay, after a bit more hunting I see the new Broker communications docs. > - https://www.bro.org/sphinx/components/broker/README.html > - https://www.bro.org/sphinx/components/broker/broker-manual.html > > I see that you can wrap the broker API with SWIG, so this is all good new. > > Anyway happen to have/make/point me to a small example python script that > maybe subscribes to all connection events (events that go into conn.long)? > > Thanks a bunch, > -Brian Wyli > > On Thu, Mar 23, 2017 at 1:40 PM, Brian Wylie > wrote: > > > Hi All, > > > > I'm fairly new to Bro and I have a question very similar to this one ' > > http://mailman.icsi.berkeley.edu/pipermail/bro/2017-January/011389.html'. > > > > Basically I want the easiest/best path to get standard Bro events (conn, > > http, dns, ssl, weird..etc) into Python. > > > > 1) Is broctl / python-broccoli the best path? > > - Note: in my testing I had to use broctl> start . in order for my > > python Connection() to work > > - If this isn't necessary and I can do the same with just running > > Bro standalone pls let me know > > > > 2) If broctl/python-broccoli IS the best path then how do I 'subscribe' to > > the standard events? > > - Is there a list of the standard events? > > - If so do I just @event with a method that has the same name as the > > event? > > > > Sorry if these are naive questions, but so far my googling/trying/testing > > has been a bit hit-miss :) > > > > Cheers, > > -Brian Wylie > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Wed Apr 5 09:46:07 2017 From: johanna at icir.org (Johanna Amann) Date: Wed, 5 Apr 2017 09:46:07 -0700 Subject: [Bro] broctl write output pcap In-Reply-To: References: Message-ID: <20170405164607.cpvy2sc2afz5gjce@wifi240.sys.ICSI.Berkeley.EDU> Hi, in theory, you can pass arbitrary flags to Bro when it is called by broctl, by setting BroArgs in broctl.cfg (see https://www.bro.org/sphinx/components/broctl/README.html). Note that writing pcap files with Bro has a few problems at the moment (I think); I think it might corrupt packages under some circumstances. It certainly is not a widely used feature and receives no testing at all. Johanna On Wed, Mar 15, 2017 at 12:00:35PM +0200, Alex Kefallonitis wrote: > I know that i can run bro -i eth0 -w .pcap . Is there a way broctl to also > write to pcap file? > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From asharma at lbl.gov Wed Apr 5 09:50:29 2017 From: asharma at lbl.gov (Aashish Sharma) Date: Wed, 5 Apr 2017 09:50:29 -0700 Subject: [Bro] NetControl configuration In-Reply-To: <20170405163051.x2f4oqrb2wik22wc@wifi240.sys.ICSI.Berkeley.EDU> References: <9d900f340345412d8c545f2ced222966@moxde9.na.bayer.cnb> <20170405163051.x2f4oqrb2wik22wc@wifi240.sys.ICSI.Berkeley.EDU> Message-ID: <20170405165027.GB31592@mac-822.local> Also not sure if worth pointing out, this is all kinds of errors and oversights: > > hook Notice::policy(n: Notice::Info) > > { > > if fox_entropy=T Then > > add n$actions[Notice::ACTION_DROP] > > add n$actions[Notice::ACTION_EMAIL]; > > } > > hook Notice::policy(n: Notice::Info) { if (n$note == FoxEntropy) ### or whatever ou are generating notice for. { add n$actions[Notice::ACTION_DROP]; add n$actions[Notice::ACTION_EMAIL]; } } Always use {} in notice action defintions. I once didn't put notice actions within {} after if conditions. I still don't forget that day because of unexpected surprises. Secondly: > > event NetControl::init() > > { > > NetControl::drop_connection (conn_id, 0, "Cyrpto Blocked") > > } Nope, you put NetControl::drop_connection in your script either associating it with a notice or some other logic. NetControl::init is to initialize - here is what I have: event NetControl::init() { local pacf_acld = NetControl::create_acld([$acld_host=127.0.0.1, $acld_port=broker_port, $acld_topic="bro/event/pacf"]); NetControl::activate(pacf_acld, 0); } then later something like this: : function drop_it(ip: addr, msg: string): bool { if (ip in drop_info && drop_info[ip]$drop_status == SUCCESS ) return T ; local result = NetControl::drop_address(ip, 20 secs, msg); print fmt ("result is %s", result); return T ; } Aashish On Wed, Apr 05, 2017 at 09:30:51AM -0700, Johanna Amann wrote: > Hi, > > The script excerpt is not quite long enough to see what exactly is going > on here (it does, for example, not show where conn_id is coming from and > how you defined it). > > Could you perhaps just post the complete script in its current state? > > Johanna > > On Thu, Mar 30, 2017 at 02:32:51PM +0000, Andrew Dellana wrote: > > Got around to adding net control to all the scripts, and now they are failing. The script is FoxIT's ransomware script. Any idea how I can get this to work? > > > > > > event NetControl::init() > > { > > NetControl::drop_connection (conn_id, 0, "Cyrpto Blocked") > > } > > > > > > hook Notice::policy(n: Notice::Info) > > { > > if fox_entropy=T Then > > add n$actions[Notice::ACTION_DROP] > > add n$actions[Notice::ACTION_EMAIL]; > > } > > > > > > > > > > error in /opt/bro/share/bro/base/init-bare.bro, lines 123-127 and /opt/bro/share/bro/base/misc/CryptoRansomCheck.bro, line 127: type clash (conn_id and conn_id) > > error in /opt/bro/share/bro/base/misc/CryptoRansomCheck.bro, line 127 and /opt/bro/share/bro/base/init-bare.bro, lines 123-127: type mismatch (conn_id and conn_id) > > error in /opt/bro/share/bro/base/misc/CryptoRansomCheck.bro, line 127: argument type mismatch in function call (NetControl::drop_connection(conn_id, 0, Cyrpto Blocked)) > > error in /opt/bro/share/bro/base/misc/CryptoRansomCheck.bro, line 128: syntax error, at or near "}" > > > > > > Freundliche Gr??e / Best regards, > > > > Andrew Dellana > > Intern > > ________________________ > > > > > > -----Original Message----- > > From: Azoff, Justin S [mailto:jazoff at illinois.edu] > > Sent: Thursday, March 16, 2017 11:08 AM > > To: Andrew Dellana > > Cc: bro at bro.org > > Subject: Re: [Bro] NetControl configuration > > > > > > > On Mar 16, 2017, at 11:04 AM, Andrew Dellana wrote: > > > > > > Yes, I do want to make the NetControl actions based on what is alerted in Notices. Can all the helpers be stored in one file and only call the helper that is needed? > > > > Yep, you can do exactly that. > > > > -- > > - Justin Azoff > > > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From espressobeanies at gmail.com Wed Apr 5 10:09:24 2017 From: espressobeanies at gmail.com (Espresso Beanies) Date: Wed, 5 Apr 2017 13:09:24 -0400 Subject: [Bro] Question about listening on multiple interfaces Message-ID: Good afternoon, I'm trying to see what would be the best way to direct Bro to listen on multiple interfaces. I currently have one interface defined in my node.cfg file but to add the second, is it best to create a separate worker or define the interface in broctl using 'broargs'? Thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170405/40859f7a/attachment.html From soehlert at es.net Wed Apr 5 16:48:23 2017 From: soehlert at es.net (Samuel Oehlert) Date: Wed, 5 Apr 2017 18:48:23 -0500 Subject: [Bro] Question about listening on multiple interfaces In-Reply-To: References: Message-ID: I've always done a separate worker in that case, which I believe is the most common way, though I'm not sure if one way is correct vs incorrect, or just common vs uncommon. -Sam On Wed, Apr 5, 2017 at 12:09 PM, Espresso Beanies wrote: > Good afternoon, > > I'm trying to see what would be the best way to direct Bro to listen on > multiple interfaces. I currently have one interface defined in my node.cfg > file but to add the second, is it best to create a separate worker or > define the interface in broctl using 'broargs'? > > Thanks, > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170405/1e7f7df2/attachment.html From bill.de.ping at gmail.com Thu Apr 6 02:59:37 2017 From: bill.de.ping at gmail.com (william de ping) Date: Thu, 6 Apr 2017 12:59:37 +0300 Subject: [Bro] minimalistic bro setup In-Reply-To: <20170405162103.27agsuge6b4wl6vm@wifi240.sys.ICSI.Berkeley.EDU> References: <20170405162103.27agsuge6b4wl6vm@wifi240.sys.ICSI.Berkeley.EDU> Message-ID: Thank you Johanna, The thing is that regardless of init-default and init-bare, there are still default plugins and analyzers that are loaded. For example, if I am not processing any TCP traffic, I do not TCP analyzer or HTTP's related plugins, and they are loaded by default.. Any ideas for that matter ? Thanks again, B On Wed, Apr 5, 2017 at 7:21 PM, Johanna Amann wrote: > You are probably looking for bare mode, which you can use by starting Bro > with the "-b" option. > > In bare mode, Bro only loads init-bare.bro, and does not load > init-default; thus basically no analyzers are activated. > > Johanna > > On Wed, Apr 05, 2017 at 03:40:37PM +0300, william de ping wrote: > > hi > > any ideas on how to turn off unwanted plugins\analyzers ? > > > > thanks > > > > On Tue, Apr 4, 2017 at 1:07 PM, william de ping > > wrote: > > > > > Hi all, > > > > > > I would like to make bro real thin by not loading all unnecessary > > > plugins\analyzers. > > > > > > I have tweaked init-bare and init-default scripts, yet when I see the > > > loaded-scripts, I see that many plugins are loaded. > > > > > > How can I turn off plugins effectively ? > > > when I edit base/bif/plugins/__load__.bro to not load ,say, FTP, I get > > > many errors that some FTP fields are not recognized and preventing the > > > cluster from running. > > > > > > I basically need only UDP and DNS events and have no need for the > moment > > > for other down stream analyzers\plugins. > > > > > > Thanks in advance > > > B > > > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170406/d9462ff1/attachment.html From johanna at icir.org Thu Apr 6 07:44:03 2017 From: johanna at icir.org (Johanna Amann) Date: Thu, 06 Apr 2017 07:44:03 -0700 Subject: [Bro] minimalistic bro setup In-Reply-To: References: <20170405162103.27agsuge6b4wl6vm@wifi240.sys.ICSI.Berkeley.EDU> Message-ID: Hi William, if you use Bro in bare mode, even though the other analyzers will be loaded, they will not be active, and thus not use any CPU time; the amount of memory they use should not be rather small (which I guess might be important if you try to get it to work on embedded devices). There currently is no easy way to prevent the shipped analyzers from loading, that I am aware of. Johanna On 6 Apr 2017, at 2:59, william de ping wrote: > Thank you Johanna, > > The thing is that regardless of init-default and init-bare, there are > still > default plugins and analyzers that are loaded. > For example, if I am not processing any TCP traffic, I do not TCP > analyzer > or HTTP's related plugins, and they are loaded by default.. > > Any ideas for that matter ? > > Thanks again, > B > > On Wed, Apr 5, 2017 at 7:21 PM, Johanna Amann > wrote: > >> You are probably looking for bare mode, which you can use by starting >> Bro >> with the "-b" option. >> >> In bare mode, Bro only loads init-bare.bro, and does not load >> init-default; thus basically no analyzers are activated. >> >> Johanna >> >> On Wed, Apr 05, 2017 at 03:40:37PM +0300, william de ping wrote: >>> hi >>> any ideas on how to turn off unwanted plugins\analyzers ? >>> >>> thanks >>> >>> On Tue, Apr 4, 2017 at 1:07 PM, william de ping >>> >>> wrote: >>> >>>> Hi all, >>>> >>>> I would like to make bro real thin by not loading all unnecessary >>>> plugins\analyzers. >>>> >>>> I have tweaked init-bare and init-default scripts, yet when I see >>>> the >>>> loaded-scripts, I see that many plugins are loaded. >>>> >>>> How can I turn off plugins effectively ? >>>> when I edit base/bif/plugins/__load__.bro to not load ,say, FTP, I >>>> get >>>> many errors that some FTP fields are not recognized and preventing >>>> the >>>> cluster from running. >>>> >>>> I basically need only UDP and DNS events and have no need for the >> moment >>>> for other down stream analyzers\plugins. >>>> >>>> Thanks in advance >>>> B >>>> >> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> From espressobeanies at gmail.com Thu Apr 6 08:54:06 2017 From: espressobeanies at gmail.com (Espresso Beanies) Date: Thu, 6 Apr 2017 11:54:06 -0400 Subject: [Bro] Question about listening on multiple interfaces In-Reply-To: References: Message-ID: I see. Yep. I confirmed I'm able to see traffic on both interfaces using the 'broargs' route. I'm not seeing much of a drop in resources or uptick in packet drops even with my node.cfg file defined to use 'lb_procs=myricom' and the second interface not being a Myricom-based NIC. Dunno if it would cause efficiency loss but I'm not able to detect it. On Wed, Apr 5, 2017 at 7:48 PM, Samuel Oehlert wrote: > I've always done a separate worker in that case, which I believe is the > most common way, though I'm not sure if one way is correct vs incorrect, or > just common vs uncommon. > > -Sam > > On Wed, Apr 5, 2017 at 12:09 PM, Espresso Beanies < > espressobeanies at gmail.com> wrote: > >> Good afternoon, >> >> I'm trying to see what would be the best way to direct Bro to listen on >> multiple interfaces. I currently have one interface defined in my node.cfg >> file but to add the second, is it best to create a separate worker or >> define the interface in broctl using 'broargs'? >> >> Thanks, >> >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170406/b4053101/attachment.html From Hafiz.Ul-Asad.1 at city.ac.uk Fri Apr 7 03:26:34 2017 From: Hafiz.Ul-Asad.1 at city.ac.uk (Ul Asad, Hafiz) Date: Fri, 7 Apr 2017 10:26:34 +0000 Subject: [Bro] Problem with installing Bro Postgresql plugin Message-ID: Hi, I have been trying to install Bro 'postgresql' plugin but have been unsuccessful. The postgresql is there on my Ubuntu 14.04 OS but apparently Bro can't find it's library and the include directory. Any help please. Asad -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170407/79332ea9/attachment.html From johanna at icir.org Fri Apr 7 09:39:51 2017 From: johanna at icir.org (Johanna Amann) Date: Fri, 07 Apr 2017 09:39:51 -0700 Subject: [Bro] Problem with installing Bro Postgresql plugin In-Reply-To: References: Message-ID: <69170466-A2D2-45BD-8E3D-7447F44A1778@icir.org> Hi Asad, please use the version available at https://github.com/0xxon/bro-postgresql and try ./configure --bro-dist=[your bro source distribution] --with-postresql-inc=`pg_config --includedir` --with-postresql-server-inc=`pg_config --includedir-server` --with-postresql-lib=`pg_config --libdir` (or just use bro-pig for the installation, which should use all this by default). Johanna On 7 Apr 2017, at 3:26, Ul Asad, Hafiz wrote: > Hi, > > I have been trying to install Bro 'postgresql' plugin but have been > unsuccessful. The postgresql is there on my Ubuntu 14.04 OS but > apparently Bro can't find it's library and the include directory. Any > help please. > > Asad > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From seth at corelight.com Fri Apr 7 09:59:14 2017 From: seth at corelight.com (Seth Hall) Date: Fri, 7 Apr 2017 12:59:14 -0400 Subject: [Bro] Problem with installing Bro Postgresql plugin In-Reply-To: <69170466-A2D2-45BD-8E3D-7447F44A1778@icir.org> References: <69170466-A2D2-45BD-8E3D-7447F44A1778@icir.org> Message-ID: <9FA7A316-E353-4FB1-9178-D3AD3EC8C54C@corelight.com> > On Apr 7, 2017, at 12:39 PM, Johanna Amann wrote: > > (or just use bro-pig for the installation, which should use all this by > default). I'm guessing that autocorrect bit Johanna here and she meant "bro-pkg". More information can be found here: http://blog.bro.org/2016/10/introducing-bro-package-manager.html .Seth -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From johanna at icir.org Fri Apr 7 11:17:03 2017 From: johanna at icir.org (Johanna Amann) Date: Fri, 07 Apr 2017 11:17:03 -0700 Subject: [Bro] Problem with installing Bro Postgresql plugin In-Reply-To: References: <69170466-A2D2-45BD-8E3D-7447F44A1778@icir.org> Message-ID: Looks like you need to install libpq-dev (and potentially postgresql-server-dev too); at the moment you probably do not have the header files installed. After that installation will probably even work without the additional flags. Johanna On 7 Apr 2017, at 10:59, Ul Asad, Hafiz wrote: > Hi, > > I am getting the error which is attached. > > Asad > > -----Original Message----- > From: Johanna Amann [mailto:johanna at icir.org] > Sent: 07 April 2017 17:40 > To: Ul Asad, Hafiz > Cc: bro at bro.org > Subject: Re: [Bro] Problem with installing Bro Postgresql plugin > > Hi Asad, > > please use the version available at > https://github.com/0xxon/bro-postgresql and try > > ./configure --bro-dist=[your bro source distribution] > --with-postresql-inc=`pg_config --includedir` > --with-postresql-server-inc=`pg_config --includedir-server` > --with-postresql-lib=`pg_config --libdir` > > (or just use bro-pig for the installation, which should use all this > by default). > > Johanna > > On 7 Apr 2017, at 3:26, Ul Asad, Hafiz wrote: > >> Hi, >> >> I have been trying to install Bro 'postgresql' plugin but have been >> unsuccessful. The postgresql is there on my Ubuntu 14.04 OS but >> apparently Bro can't find it's library and the include directory. Any >> help please. >> >> Asad >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From espressobeanies at gmail.com Fri Apr 7 12:36:02 2017 From: espressobeanies at gmail.com (Espresso Beanies) Date: Fri, 7 Apr 2017 15:36:02 -0400 Subject: [Bro] Question regarding manager and proxy defined in node.cfg Message-ID: Hi, I've seen documentation where node.cfg files are defined as: [manager] type=manager host=127.0.0.1 interface=eth0 [proxy] type=manager host=127.0.0.1 interface=eth0 And I've also seen other documentation defined that excludes the interface: [manager] type=manager host=127.0.0.1 [proxy] type=manager host=127.0.0.1 If I'm running Bro on a local machine, no cluster configuration, etc, what effect does defining the interface type for the proxy and manager have over not defining it at all? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170407/06a407b4/attachment.html From jan.grashoefer at gmail.com Sun Apr 9 02:06:54 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Sun, 9 Apr 2017 11:06:54 +0200 Subject: [Bro] dpdk In-Reply-To: References: Message-ID: > Any idea if this will be supported? I can not find any reference in the > past year indicating this one way or another. >From what I've read so far (e.g., https://www.napatech.com/dpdk-packet-capture-pdump/), I wouldn't expect major performance boosts. Therefore I am curios: Where so you see the benefits of using dpdk? Jan From ed.sealing at sealingtech.org Sun Apr 9 07:14:46 2017 From: ed.sealing at sealingtech.org (Ed Sealing) Date: Sun, 9 Apr 2017 10:14:46 -0400 Subject: [Bro] dpdk In-Reply-To: References: Message-ID: On Sun, Apr 9, 2017 at 5:06 AM, Jan Grash?fer wrote: > > Any idea if this will be supported? I can not find any reference in the > > past year indicating this one way or another. > > >From what I've read so far (e.g., > https://www.napatech.com/dpdk-packet-capture-pdump/), I wouldn't expect > major performance boosts. Therefore I am curios: Where so you see the > benefits of using dpdk? > > I believe there would be some benefits in the ability to run high-speed packet capture in VMs or Containers that are hosted on a cloud management system (CMS). The world of NFV and service function chaining (which encompasses IDSs such as Bro) often relies on DPDK applications. Many of the CMS providers (e.g. Openstack, Kubernetes, etc) rely on DPDK-enabled vSwitches such as OVS and VPP for accelerated packet distribution. A DPDK-enabled Bro would be able to take advantage of bypassing the VM kernel as well as reading the packets directly from the vSwitches shared memory (some possible security concerns there). A brief overview of how this would work with openvswitch is at [1]. Other potential benefits areas for the virtual space are when using SR-IOV, which have different drivers (ixgbevf & i40evf) that aren't widely supported by zero-copy technologies (NOTE: netmap has recently included support for ixgbevf, and packet-bricks may be able to read from virtio devices and fanout, but I haven't tested yet). I don't know that these benefits are enough to justify the amount of development work it would take to implement and maintain a DPDK packet acquisition plugin. Just throwing out an answer to the question. :-) [1] https://software.intel.com/en-us/articles/configure-vhost-user-multiqueue-for-ovs-with-dpdk . ~Ed > Jan > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170409/2d23f653/attachment.html From jan.grashoefer at gmail.com Mon Apr 10 02:26:37 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Mon, 10 Apr 2017 11:26:37 +0200 Subject: [Bro] dpdk In-Reply-To: References: Message-ID: <8e948a6b-43a0-e3c7-6885-5d77962a96ab@gmail.com> Hi Ed, thanks a lot for your detailed explanation! > I believe there would be some benefits in the ability to run high-speed > packet capture in VMs or Containers that are hosted on a cloud management > system (CMS). The world of NFV and service function chaining (which > encompasses IDSs such as Bro) often relies on DPDK applications. With virtualization in mind, using DPDK for packet acquisition seems to make sense. > I don't know that these benefits are enough to justify the amount of > development work it would take to implement and maintain a DPDK packet > acquisition plugin. Just throwing out an answer to the question. :-) At least it is worth a POC I think :) Jan From shirkdog.bsd at gmail.com Mon Apr 10 05:14:31 2017 From: shirkdog.bsd at gmail.com (Michael Shirk) Date: Mon, 10 Apr 2017 08:14:31 -0400 Subject: [Bro] dpdk In-Reply-To: <8e948a6b-43a0-e3c7-6885-5d77962a96ab@gmail.com> References: <8e948a6b-43a0-e3c7-6885-5d77962a96ab@gmail.com> Message-ID: DPDK enable apps are apparently something that can be built on FreeBSD, just need the port as recommended by Intel. I have not been able to play with this, but it would be nice to have options in addition to netmap for fast packet acquisition on FreeBSD. -- Michael Shirk Daemon Security, Inc. http://www.daemon-security.com On Apr 10, 2017 5:28 AM, "Jan Grash?fer" wrote: > Hi Ed, > > thanks a lot for your detailed explanation! > > > I believe there would be some benefits in the ability to run high-speed > > packet capture in VMs or Containers that are hosted on a cloud management > > system (CMS). The world of NFV and service function chaining (which > > encompasses IDSs such as Bro) often relies on DPDK applications. > > With virtualization in mind, using DPDK for packet acquisition seems to > make sense. > > > I don't know that these benefits are enough to justify the amount of > > development work it would take to implement and maintain a DPDK packet > > acquisition plugin. Just throwing out an answer to the question. :-) > > At least it is worth a POC I think :) > > Jan > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170410/beeef30b/attachment.html From pssunu6 at gmail.com Mon Apr 10 08:21:27 2017 From: pssunu6 at gmail.com (ps sunu) Date: Mon, 10 Apr 2017 20:51:27 +0530 Subject: [Bro] dhcp log Message-ID: Hi, i created a bro which will record "client" field and write it in dhcp.log,the problem the recording part is working but its not writing into dhcp file , its creating orig_hostname but its is blank any one help me below my script module TrackCONN; export { global host_name_user1: table[addr] of string &synchronized &write_expire=7day; redef record DHCP::Info += { orig_hostname: string &log &optional; # take from dhcp hostname and kerberos host }; } event KRB::log_krb (rec: KRB::Info) { if(rec?$client) { host_name_user1[rec$id$orig_h] = rec$client; #print host_name_user1[rec$id$orig_h]; } } Regards, Sunu event dhcp_ack(c: connection, msg: dhcp_msg, mask: addr, router: dhcp_router_list, lease: interval, serv_addr: addr, host_name: string) { if ( c$dhcp$id$orig_h in TrackCONN::host_name_user1 ) c$dhcp$orig_hostname = TrackCONN::host_name_user1[c$dhcp$id$orig_h]; print host_name_user1[c$id$orig_h]; } -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170410/f23e4d4f/attachment.html From fatema.bannatwala at gmail.com Mon Apr 10 08:54:42 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Mon, 10 Apr 2017 11:54:42 -0400 Subject: [Bro] dhcp log In-Reply-To: References: Message-ID: Could you try following in place of the original dhcp_ack event: event dhcp_ack(c: connection, msg: dhcp_msg, mask: addr, router: dhcp_router_list, lease: interval, serv_addr: addr, host_name: string) { local info: DHCP::Info; if ( c$dhcp$id$orig_h in TrackCONN::host_name_user1 ) { info$orig_hostname = TrackCONN::host_name_user1[c$dhcp$id$orig_h]; print host_name_user1[c$id$orig_h]; } } On Mon, Apr 10, 2017 at 11:21 AM, ps sunu wrote: > Hi, > i created a bro which will record "client" field and write it in > dhcp.log,the problem the recording part is working but its not writing into > dhcp file , its creating orig_hostname but its is blank any one help me > > below my script > > module TrackCONN; > > > export { > global host_name_user1: table[addr] of string &synchronized &write_expire=7day; > > > redef record DHCP::Info += { > > orig_hostname: string &log &optional; # take from dhcp hostname and kerberos host > > > }; > } > > > event KRB::log_krb (rec: KRB::Info) > { > if(rec?$client) { > host_name_user1[rec$id$orig_h] = rec$client; > #print host_name_user1[rec$id$orig_h]; > } > > > > } > > > Regards, > > Sunu > > > event dhcp_ack(c: connection, msg: dhcp_msg, mask: addr, router: dhcp_router_list, lease: interval, serv_addr: addr, host_name: string) > { > > if ( c$dhcp$id$orig_h in TrackCONN::host_name_user1 ) > c$dhcp$orig_hostname = TrackCONN::host_name_user1[c$dhcp$id$orig_h]; > print host_name_user1[c$id$orig_h]; > } > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170410/b2ba31f4/attachment.html From pssunu6 at gmail.com Mon Apr 10 09:13:10 2017 From: pssunu6 at gmail.com (ps sunu) Date: Mon, 10 Apr 2017 21:43:10 +0530 Subject: [Bro] dhcp log In-Reply-To: References: Message-ID: no change same output On Mon, Apr 10, 2017 at 9:24 PM, fatema bannatwala < fatema.bannatwala at gmail.com> wrote: > Could you try following in place of the original dhcp_ack event: > > event dhcp_ack(c: connection, msg: dhcp_msg, mask: addr, router: dhcp_router_list, lease: interval, serv_addr: addr, host_name: string) > { > local info: DHCP::Info; > if ( c$dhcp$id$orig_h in TrackCONN::host_name_user1 ) > { info$orig_hostname = TrackCONN::host_name_user1[c$dhcp$id$orig_h]; > print host_name_user1[c$id$orig_h]; > } > } > > > On Mon, Apr 10, 2017 at 11:21 AM, ps sunu wrote: > >> Hi, >> i created a bro which will record "client" field and write it in >> dhcp.log,the problem the recording part is working but its not writing into >> dhcp file , its creating orig_hostname but its is blank any one help me >> >> below my script >> >> module TrackCONN; >> >> >> export { >> global host_name_user1: table[addr] of string &synchronized &write_expire=7day; >> >> >> redef record DHCP::Info += { >> >> orig_hostname: string &log &optional; # take from dhcp hostname and kerberos host >> >> >> }; >> } >> >> >> event KRB::log_krb (rec: KRB::Info) >> { >> if(rec?$client) { >> host_name_user1[rec$id$orig_h] = rec$client; >> #print host_name_user1[rec$id$orig_h]; >> } >> >> >> >> } >> >> >> Regards, >> >> Sunu >> >> >> event dhcp_ack(c: connection, msg: dhcp_msg, mask: addr, router: dhcp_router_list, lease: interval, serv_addr: addr, host_name: string) >> { >> >> if ( c$dhcp$id$orig_h in TrackCONN::host_name_user1 ) >> c$dhcp$orig_hostname = TrackCONN::host_name_user1[c$dhcp$id$orig_h]; >> print host_name_user1[c$id$orig_h]; >> } >> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170410/e93c2290/attachment-0001.html From pssunu6 at gmail.com Mon Apr 10 10:41:10 2017 From: pssunu6 at gmail.com (ps sunu) Date: Mon, 10 Apr 2017 23:11:10 +0530 Subject: [Bro] dhcp log In-Reply-To: References: Message-ID: no print output . and its blank dhcp.log #open 2017-04-10-10-37-27 #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p mac assigned_ip lease_time trans_id orig_hostname resp_hostname #types time string addr port addr port string addr interval count string string 1491343946.444166 CHcwRb1IXBXBSql2Jk 255.255.255.255 68 10.16.80.1 67 4c:34:88:02:43:d3 255.255.255.255 0.000000 3620830950 - - 1491343946.444199 CHcwRb1IXBXBSql2Jk 255.255.255.255 68 10.16.80.1 67 4c:34:88:02:43:d3 255.255.255.255 0.000000 3620830950 - - On Mon, Apr 10, 2017 at 9:43 PM, ps sunu wrote: > no change same output > > On Mon, Apr 10, 2017 at 9:24 PM, fatema bannatwala < > fatema.bannatwala at gmail.com> wrote: > >> Could you try following in place of the original dhcp_ack event: >> >> event dhcp_ack(c: connection, msg: dhcp_msg, mask: addr, router: dhcp_router_list, lease: interval, serv_addr: addr, host_name: string) >> { >> local info: DHCP::Info; >> if ( c$dhcp$id$orig_h in TrackCONN::host_name_user1 ) >> { info$orig_hostname = TrackCONN::host_name_user1[c$dhcp$id$orig_h]; >> print host_name_user1[c$id$orig_h]; >> } >> } >> >> >> On Mon, Apr 10, 2017 at 11:21 AM, ps sunu wrote: >> >>> Hi, >>> i created a bro which will record "client" field and write it in >>> dhcp.log,the problem the recording part is working but its not writing into >>> dhcp file , its creating orig_hostname but its is blank any one help me >>> >>> below my script >>> >>> module TrackCONN; >>> >>> >>> export { >>> global host_name_user1: table[addr] of string &synchronized &write_expire=7day; >>> >>> >>> redef record DHCP::Info += { >>> >>> orig_hostname: string &log &optional; # take from dhcp hostname and kerberos host >>> >>> >>> }; >>> } >>> >>> >>> event KRB::log_krb (rec: KRB::Info) >>> { >>> if(rec?$client) { >>> host_name_user1[rec$id$orig_h] = rec$client; >>> #print host_name_user1[rec$id$orig_h]; >>> } >>> >>> >>> >>> } >>> >>> >>> Regards, >>> >>> Sunu >>> >>> >>> event dhcp_ack(c: connection, msg: dhcp_msg, mask: addr, router: dhcp_router_list, lease: interval, serv_addr: addr, host_name: string) >>> { >>> >>> if ( c$dhcp$id$orig_h in TrackCONN::host_name_user1 ) >>> c$dhcp$orig_hostname = TrackCONN::host_name_user1[c$dhcp$id$orig_h]; >>> print host_name_user1[c$id$orig_h]; >>> } >>> >>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170410/b602bbda/attachment.html From pssunu6 at gmail.com Mon Apr 10 12:31:31 2017 From: pssunu6 at gmail.com (ps sunu) Date: Tue, 11 Apr 2017 01:01:31 +0530 Subject: [Bro] dhcp.log Message-ID: Hi, i need to copy kerberos.log client field value logs into dhcp.log new field example hos_test is it possible ? regards sunu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170411/0f52e44d/attachment.html From pssunu6 at gmail.com Tue Apr 11 01:10:31 2017 From: pssunu6 at gmail.com (ps sunu) Date: Tue, 11 Apr 2017 13:40:31 +0530 Subject: [Bro] (no subject) Message-ID: Hi, i need known-device.log "dhcp_host_name" field into dhcp.log below my code which will write dhcp_host_name into dhcp.log but its giving error many way i tried many ways but not find solution and , some times its not giving error but its not writing dhcp_host_name into dhcp.log ##! Tracks MAC address with hostnames seen in DHCP traffic. They are logged into ##! ``devices.log``. @load /opt/bro/share/bro/policy/misc/known-devices.bro module Known1; export { redef record DHCP::Info += { ## The value of the DHCP host name option, if seen. dhcp_host_name: string &log &optional; }; } event dhcp_request(c: connection, msg: dhcp_msg, req_addr: addr, serv_addr: addr, host_name: string) { if ( msg$h_addr == "" ) return; if ( msg$h_addr !in known_devices ) { add known_devices[msg$h_addr]; c$dhcp$$dhcp_host_name = host_name; } } event dhcp_inform(c: connection, msg: dhcp_msg, host_name: string) { if ( msg$h_addr == "" ) return; if ( msg$h_addr !in known_devices ) { add known_devices[msg$h_addr]; c$dhcp$$dhcp_host_name = host_name; } } -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170411/f123dfb3/attachment.html From al.kefallonitis at gmail.com Tue Apr 11 04:10:08 2017 From: al.kefallonitis at gmail.com (Alex Kefallonitis) Date: Tue, 11 Apr 2017 14:10:08 +0300 Subject: [Bro] Bro sqli + xss sans paper Message-ID: I am trying to add the two scripts for sqli and xss from this paper https://www.sans.org/reading-room/whitepapers/detection/web-application-attack-analysis-bro-ids-34042 but i get this error HTTP::c$http$first_chunk no such a field in record... Anyone knows what is happening? Thanks in advanced. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170411/86907448/attachment.html From pssunu6 at gmail.com Tue Apr 11 05:21:28 2017 From: pssunu6 at gmail.com (ps sunu) Date: Tue, 11 Apr 2017 17:51:28 +0530 Subject: [Bro] dhcp_host_name field Message-ID: Hi, I need to copy "host_name" output into dhcp.log "dhcp_host_name " below my script please help me http://try.bro.org/#/trybro/saved/137433 my script ##! Tracks MAC address with hostnames seen in DHCP traffic. They are logged into ##! ``devices.log``. @load policy/misc/known-devices module Known; export { redef record DHCP::Info += { ## The value of the DHCP host name option, if seen. dhcp_host_name: string &log &optional; }; } event dhcp_request(c: connection, msg: dhcp_msg, req_addr: addr, serv_addr: addr, host_name: string) { if ( host_name != "" ) if (c?$dhcp) { c$dhcp$dhcp_host_name = host_name; } # c$dhcp = info; #print host_name; #print c$dhcp$dhcp_host_name; #print c$dhcp; # Log::write(Known::DEVICES_LOG, [$ts=network_time(), $mac=msg$h_addr, $dhcp_host_name=host_name]); } -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170411/15d62edf/attachment.html From finid at vivaldi.net Wed Apr 12 07:05:55 2017 From: finid at vivaldi.net (LinuxBSDos.com) Date: Wed, 12 Apr 2017 09:05:55 -0500 Subject: [Bro] Regarding Broctl cron Message-ID: <68ce58a13018e134fdf87431ed4d300b@vivaldi.net> Hello: I've set up a cron job for "broctl cron", and verified that the cron job runs every 5 minutes. To test that the script works, I stop and watched for it to be restart by the script, but it's not happening. I'm aware that the docs says the "main purpose of the BroControl cron command is to check for Bro nodes that have crashed, and to restart them". Though mine didn't crash, I'm expecting that the script will at least notice that it's not running and restart it. Isn't that how it's supposed t work? The installation, by the way, is on Ubuntu 16.04 Thanks, -finid- Sent from my containerized Linux desktop ---------------------------------------- From jazoff at illinois.edu Wed Apr 12 07:34:19 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 12 Apr 2017 14:34:19 +0000 Subject: [Bro] Regarding Broctl cron In-Reply-To: <68ce58a13018e134fdf87431ed4d300b@vivaldi.net> References: <68ce58a13018e134fdf87431ed4d300b@vivaldi.net> Message-ID: <959D3168-05B2-4DEC-A201-A26BF95748A1@illinois.edu> > On Apr 12, 2017, at 9:05 AM, LinuxBSDos.com wrote: > > Hello: > > I've set up a cron job for "broctl cron", and verified that the cron job > runs every 5 minutes. To test that the script works, I stop and watched > for it to be restart by the script, but it's not happening. > > I'm aware that the docs says the "main purpose of the BroControl cron > command is to check for Bro nodes that have crashed, and to restart > them". Though mine didn't crash, I'm expecting that the script will at > least notice that it's not running and restart it. > > Isn't that how it's supposed t work? If you stopped bro on purpose it will not restart it. If you want to test the restart functionality you will need to kill one of the bro processes. -- - Justin Azoff From finid at vivaldi.net Wed Apr 12 11:45:21 2017 From: finid at vivaldi.net (LinuxBSDos.com) Date: Wed, 12 Apr 2017 13:45:21 -0500 Subject: [Bro] Bro and GeoIP Message-ID: <2d1013f707467ed9b0de2443b4525cdb@vivaldi.net> Hello, If I installed Bro using the package manager, made sure that the GeoIP databases are in the right place, what else do I need to make it work, or does Bro need to be compiled from source for it to have support for GeoIP? I've done the test at [1] and it returned the error. Thanks, https://www.bro.org/sphinx/frameworks/geoip.html#geolocation -- -finid- Sent from my containerized Linux desktop ---------------------------------------- From briford.wylie at gmail.com Wed Apr 12 14:11:23 2017 From: briford.wylie at gmail.com (Brian Wylie) Date: Wed, 12 Apr 2017 15:11:23 -0600 Subject: [Bro] Yara integration with Bro 2.5 In-Reply-To: References: Message-ID: Erik has a good point about overhead. So for offloading (and because I love python) I threw together the Bro to Python repo and added a yara example. Might be useful if you like Python.. shrug...donno... just throwing it out there :) - https://github.com/Kitware/BroThon On Wed, Apr 5, 2017 at 7:02 AM, erik clark wrote: > Er, doesnt this come with massive overhead? Also, file inspection rules > are non-trivial. Given the number of files that bro processes, it seems > that on anything other than a very tiny link this would cause giant > problems... > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170412/5d6d6035/attachment.html From finid at vivaldi.net Wed Apr 12 16:41:46 2017 From: finid at vivaldi.net (LinuxBSDos.com) Date: Wed, 12 Apr 2017 18:41:46 -0500 Subject: [Bro] Specifying scripts for Bro to load Message-ID: Hi: On this documentation page [1], it says "all" can be used to specify that Bro "perform all the default analysis that?s available". However when I attach "all" to the Bro command, like "bro -i eth0 all" is exits with an error: "fatal error: can't find all". I know that "local" is used to load site-specific scripts, what can I use to load all available scripts, as indicated in the docs? [1] https://www.bro.org/sphinx/quickstart/index.html#bro-as-a-command-line-utility Thanks, -finid- Sent from my containerized Linux desktop ---------------------------------------- From pssunu6 at gmail.com Wed Apr 12 22:49:52 2017 From: pssunu6 at gmail.com (ps sunu) Date: Thu, 13 Apr 2017 11:19:52 +0530 Subject: [Bro] intel.log extra log Message-ID: Hi, i need to generate intel log content into separate logs , i generated except one field i am not able to get "source" field into separate log, any idea tp do this my main file @load frameworks/intel/seen redef Intel::read_files += { fmt("%s/intel-1.dat", @DIR) }; @load ./field.bro #redef LogAscii::use_json=T; event Intel::log_intel (rec: Intel::Info) { # Log::create_stream(Factor::LOG, [$columns=Factor::Info, $path="intel_trigger"]); if ( rec$seen$where == HTTP::IN_HOST_HEADER ) { # print "ssss",rec$seen; Log::write(Match::LOG,[$ts=network_time(),$uid=rec$seen$uid,$id=rec$seen$conn$id,$seen_indicator=rec$seen$indicator,$seen_indicator_type=rec$seen$indicator_type,$seen_where=rec$seen$where,$seen_node=rec$seen$node,$matched=rec$seen$indicator_type ]); print "ssssssss",rec$seen; } } event bro_init () { Log::create_stream(Match::LOG, [$columns=Match::Info, $path="intel_tech"]); } field.bro module Match; export { # Append the value LOG to the Log::ID enumerable. redef enum Log::ID += { LOG }; type Type: enum { ## An IP address. ADDR, ## A complete URL without the prefix ``"http://"``. URL, ## Software name. SOFTWARE, ## Email address. EMAIL, ## DNS domain name. DOMAIN, ## A user name. USER_NAME, ## File hash which is non-hash type specific. It's up to the ## user to query for any relevant hash types. FILE_HASH, ## File name. Typically with protocols with definite ## indications of a file name. FILE_NAME, ## Certificate SHA-1 hash. CERT_HASH, ## Public key MD5 hash. (SSH server host keys are a good example.) PUBKEY_HASH, }; type Where: enum { ## A catchall value to represent data of unknown provenance. IN_ANYWHERE, }; # Define a new type called Factor::Info. type Info: record { ts: time &log; uid : string &log; id: conn_id &log; seen_indicator: string &log; seen_indicator_type: Type &log &optional; seen_where : Where &log; seen_node : string &log; matched:Type &log &optional; }; } i need intel-1.dat file www.reddit.com Intel::DOMAIN *my_special_source -->> into source field * *http://try.bro.org/#/trybro/saved/138000 * -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170413/4ad31a42/attachment.html From pssunu6 at gmail.com Wed Apr 12 23:55:11 2017 From: pssunu6 at gmail.com (ps sunu) Date: Thu, 13 Apr 2017 12:25:11 +0530 Subject: [Bro] intel.log extra log In-Reply-To: References: Message-ID: i solved the problem On Thu, Apr 13, 2017 at 11:19 AM, ps sunu wrote: > Hi, > > i need to generate intel log content into separate logs > , i generated except one field > i am not able to get "source" field into separate log, any idea tp do this > > > my main file > @load frameworks/intel/seen > > redef Intel::read_files += { > fmt("%s/intel-1.dat", @DIR) > }; > > @load ./field.bro > #redef LogAscii::use_json=T; > > > event Intel::log_intel (rec: Intel::Info) > > > { > # Log::create_stream(Factor::LOG, [$columns=Factor::Info, > $path="intel_trigger"]); > if ( rec$seen$where == HTTP::IN_HOST_HEADER ) > { > # print "ssss",rec$seen; > > Log::write(Match::LOG,[$ts=network_time(),$uid=rec$seen$ > uid,$id=rec$seen$conn$id,$seen_indicator=rec$seen$ > indicator,$seen_indicator_type=rec$seen$indicator_type,$ > seen_where=rec$seen$where,$seen_node=rec$seen$node,$ > matched=rec$seen$indicator_type ]); > print "ssssssss",rec$seen; > > > } > } > event bro_init () > > { > Log::create_stream(Match::LOG, [$columns=Match::Info, > $path="intel_tech"]); > > > } > > > field.bro > module Match; > > export { > # Append the value LOG to the Log::ID enumerable. > redef enum Log::ID += { LOG }; > > type Type: enum { > ## An IP address. > ADDR, > ## A complete URL without the prefix ``"http://"``. > URL, > ## Software name. > SOFTWARE, > ## Email address. > EMAIL, > ## DNS domain name. > DOMAIN, > ## A user name. > USER_NAME, > ## File hash which is non-hash type specific. It's up to > the > ## user to query for any relevant hash types. > FILE_HASH, > ## File name. Typically with protocols with definite > ## indications of a file name. > FILE_NAME, > ## Certificate SHA-1 hash. > CERT_HASH, > ## Public key MD5 hash. (SSH server host keys are a good > example.) > PUBKEY_HASH, > }; > > type Where: enum { > ## A catchall value to represent data of unknown > provenance. > IN_ANYWHERE, > }; > > > # Define a new type called Factor::Info. > type Info: record { > ts: time &log; > uid : string &log; > id: conn_id &log; > seen_indicator: string &log; > seen_indicator_type: Type &log &optional; > seen_where : Where &log; > seen_node : string &log; > matched:Type &log &optional; > > > }; > } > > > i need intel-1.dat file > > www.reddit.com Intel::DOMAIN *my_special_source -->> into source > field * > > > *http://try.bro.org/#/trybro/saved/138000 > * > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170413/0b8124bd/attachment-0001.html From finid at vivaldi.net Thu Apr 13 10:52:03 2017 From: finid at vivaldi.net (LinuxBSDos.com) Date: Thu, 13 Apr 2017 12:52:03 -0500 Subject: [Bro] Bro and GeoIP support Message-ID: Hello: I know that Bro has support for MaxMind's Legacy GeoIP database, which it automatically builds support for if the libgeoip library is installed. Is that also true for the newer mmdb database? If not, what's the process of getting Bro to support the mmdb database? Thanks, -finid- Sent from my containerized Linux desktop ---------------------------------------- From dave.a.florek at gmail.com Thu Apr 13 12:38:49 2017 From: dave.a.florek at gmail.com (Dave Florek) Date: Thu, 13 Apr 2017 15:38:49 -0400 Subject: [Bro] Issue with Bro and libpcap during compile Message-ID: Good afternoon, I'm curious to see if anyone in the Bro community has been successful at installing Bro from source using the initial libpcap files bundled with CentOS 7. No matter if I specify ./configure --with-pcap=/usr/local/ or ./configure --with-pcap=/usr/lib64/ after installing libpcap-devel, I'm still unable to get Bro to compile: ... > -- Looking for sigset > -- Looking for sigset - found > -- Performing Test DO_SOCK_DECL > -- Performing Test DO_SOCK_DECL - Failed > -- Performing Test SYSLOG_INT > -- Performing Test SYSLOG_INT - Failed > -- Looking for include file pcap-int.h > -- Looking for include file pcap-int.h - not found > -- Looking for pcap_freecode > -- Looking for pcap_freecode - not found > -- No implementation for pcap_freecode() > -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_HAS_ERROR_PARAMETER > -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_HAS_ERROR_PARAMETER - > Failed > -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_NO_ERROR_PARAMETER > -- Performing Test LIBPCAP_PCAP_COMPILE_NOPCAP_NO_ERROR_PARAMETER - Failed > CMake Error at cmake/PCAPTests.cmake:58 (message): > Can't determine if pcap_compile_nopcap takes an error parameter > Call Stack (most recent call first): > CMakeLists.txt:164 (include) > > -- Configuring incomplete, errors occurred! > See also "/root/bro-2.5/build/CMakeFiles/CMakeOutput.log". > See also "/root/bro-2.5/build/CMakeFiles/CMakeError.log". I tried several possible solutions from older articles scattered across the web including the exporting LIBS and LDFLAGS variables prior to compiling without success: http://mailman.icsi.berkeley.edu/pipermail/bro/2014-August/ 007404.html Is there a minimum libpcap version that Bro will only support / compile against? Thank you, -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170413/d713bd60/attachment.html From jlay at slave-tothe-box.net Fri Apr 14 08:34:15 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 14 Apr 2017 09:34:15 -0600 Subject: [Bro] Regarding Broctl cron In-Reply-To: <959D3168-05B2-4DEC-A201-A26BF95748A1@illinois.edu> References: <68ce58a13018e134fdf87431ed4d300b@vivaldi.net> <959D3168-05B2-4DEC-A201-A26BF95748A1@illinois.edu> Message-ID: <246e90e7ade1a0330e2c79c30acea993@localhost> On 2017-04-12 08:34, Azoff, Justin S wrote: >> On Apr 12, 2017, at 9:05 AM, LinuxBSDos.com wrote: >> >> Hello: >> >> I've set up a cron job for "broctl cron", and verified that the cron >> job >> runs every 5 minutes. To test that the script works, I stop and >> watched >> for it to be restart by the script, but it's not happening. >> >> I'm aware that the docs says the "main purpose of the BroControl cron >> command is to check for Bro nodes that have crashed, and to restart >> them". Though mine didn't crash, I'm expecting that the script will at >> least notice that it's not running and restart it. >> >> Isn't that how it's supposed t work? > > If you stopped bro on purpose it will not restart it. If you want to > test the restart functionality you will need to kill one of the bro > processes. I'm in this boat as well: Apr 14 15:08:56 kernel: [1371688.768856] bro invoked oom-killer: gfp_mask=0x24280ca, order=0, oom_score_adj=0 Apr 14 15:09:09 kernel: [1371700.888321] bro invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0 Apr 14 15:29:06 kernel: [1372898.864738] bro invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0 Apr 14 15:29:15 kernel: [1372907.790049] bro invoked oom-killer: gfp_mask=0x24280ca, order=0, oom_score_adj=0 Name Type Host Status Pid Started manager manager localhost running 117467 13 Apr 21:38:46 proxy-1 proxy localhost running 117509 13 Apr 21:38:48 worker-1-1 worker localhost crashed worker-1-2 worker localhost running 117778 13 Apr 21:38:49 worker-1-3 worker localhost crashed worker-1-4 worker localhost crashed worker-1-5 worker localhost running 117777 13 Apr 21:38:49 worker-1-6 worker localhost crashed worker-2-1 worker localhost running 117787 13 Apr 21:38:49 worker-2-2 worker localhost running 117775 13 Apr 21:38:49 worker-2-3 worker localhost running 117783 13 Apr 21:38:49 worker-2-4 worker localhost running 117779 13 Apr 21:38:49 worker-3-1 worker localhost running 117784 13 Apr 21:38:49 worker-3-2 worker localhost running 117780 13 Apr 21:38:49 worker-3-3 worker localhost running 117789 13 Apr 21:38:49 worker-3-4 worker localhost running 117788 13 Apr 21:38:49 worker-3-5 worker localhost running 117786 13 Apr 21:38:49 worker-3-6 worker localhost running 117790 13 Apr 21:38:49 and from crontab: 0-59/5 * * * * /opt/bro/bin/broctl cron Cron job just isn't restarting these....manually running broctl cron works though. James From jazoff at illinois.edu Fri Apr 14 08:46:24 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 14 Apr 2017 15:46:24 +0000 Subject: [Bro] Regarding Broctl cron In-Reply-To: <246e90e7ade1a0330e2c79c30acea993@localhost> References: <68ce58a13018e134fdf87431ed4d300b@vivaldi.net> <959D3168-05B2-4DEC-A201-A26BF95748A1@illinois.edu> <246e90e7ade1a0330e2c79c30acea993@localhost> Message-ID: <9B4173FA-EBE8-43C7-BA7B-A67D0F0D9671@illinois.edu> > On Apr 14, 2017, at 10:34 AM, James Lay wrote: > > and from crontab: > 0-59/5 * * * * /opt/bro/bin/broctl cron > > Cron job just isn't restarting these....manually running broctl cron > works though. Is the cron job definitely running? Is that in a users crontab or something in /etc/? If it's a system wide one in etc you'll need a user in there We use this: */5 * * * * root /bro/bin/broctl cron -- - Justin Azoff From dnthayer at illinois.edu Fri Apr 14 09:22:53 2017 From: dnthayer at illinois.edu (Daniel Thayer) Date: Fri, 14 Apr 2017 11:22:53 -0500 Subject: [Bro] Regarding Broctl cron In-Reply-To: <9B4173FA-EBE8-43C7-BA7B-A67D0F0D9671@illinois.edu> References: <68ce58a13018e134fdf87431ed4d300b@vivaldi.net> <959D3168-05B2-4DEC-A201-A26BF95748A1@illinois.edu> <246e90e7ade1a0330e2c79c30acea993@localhost> <9B4173FA-EBE8-43C7-BA7B-A67D0F0D9671@illinois.edu> Message-ID: <554914c8-feff-8894-ab7a-e266dd5e8126@illinois.edu> On 4/14/17 10:46 AM, Azoff, Justin S wrote: > >> On Apr 14, 2017, at 10:34 AM, James Lay wrote: >> >> and from crontab: >> 0-59/5 * * * * /opt/bro/bin/broctl cron >> >> Cron job just isn't restarting these....manually running broctl cron >> works though. > > Is the cron job definitely running? Is that in a users crontab or something in /etc/? If it's a system wide one in etc you'll need a user in there > > We use this: > > */5 * * * * root /bro/bin/broctl cron > Another thing to check is to run this command: broctl cron '?' The output should be: cron enabled If it says "disabled", then "broctl cron" won't do anything. From jlay at slave-tothe-box.net Fri Apr 14 10:37:06 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 14 Apr 2017 11:37:06 -0600 Subject: [Bro] Quick question on SitePolicyStandalone Message-ID: <64165fcc22946590f4d8efbbb5b046e0@localhost> So ever since my 2.4 to 2.5 upgrade I get the below when starting: Warning: the SitePolicyStandalone option is deprecated (use SitePolicyScripts instead). Any way to disable this? Thank you. James From dopheide at gmail.com Fri Apr 14 10:56:40 2017 From: dopheide at gmail.com (Mike Dopheide) Date: Fri, 14 Apr 2017 10:56:40 -0700 Subject: [Bro] Quick question on SitePolicyStandalone In-Reply-To: <64165fcc22946590f4d8efbbb5b046e0@localhost> References: <64165fcc22946590f4d8efbbb5b046e0@localhost> Message-ID: comment it out in etc/broctl.cfg :) On Fri, Apr 14, 2017 at 10:37 AM, James Lay wrote: > So ever since my 2.4 to 2.5 upgrade I get the below when starting: > > Warning: the SitePolicyStandalone option is deprecated (use > SitePolicyScripts instead). > > Any way to disable this? Thank you. > > James > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170414/6f6bc23c/attachment.html From jlay at slave-tothe-box.net Fri Apr 14 11:09:00 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 14 Apr 2017 12:09:00 -0600 Subject: [Bro] Quick question on SitePolicyStandalone In-Reply-To: References: <64165fcc22946590f4d8efbbb5b046e0@localhost> Message-ID: <2f203e4cb75d453a6c0edff1f3df2416@localhost> Where were you when I upgraded Mike :P Thanks...quick answer and just what I needed. James On 2017-04-14 11:56, Mike Dopheide wrote: > comment it out in etc/broctl.cfg :) > > On Fri, Apr 14, 2017 at 10:37 AM, James Lay > wrote: > >> So ever since my 2.4 to 2.5 upgrade I get the below when starting: >> >> Warning: the SitePolicyStandalone option is deprecated (use >> SitePolicyScripts instead). >> >> Any way to disable this? Thank you. >> >> James >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro [1] > > > > Links: > ------ > [1] http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jlay at slave-tothe-box.net Fri Apr 14 11:18:48 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 14 Apr 2017 12:18:48 -0600 Subject: [Bro] Regarding Broctl cron In-Reply-To: <554914c8-feff-8894-ab7a-e266dd5e8126@illinois.edu> References: <68ce58a13018e134fdf87431ed4d300b@vivaldi.net> <959D3168-05B2-4DEC-A201-A26BF95748A1@illinois.edu> <246e90e7ade1a0330e2c79c30acea993@localhost> <9B4173FA-EBE8-43C7-BA7B-A67D0F0D9671@illinois.edu> <554914c8-feff-8894-ab7a-e266dd5e8126@illinois.edu> Message-ID: On 2017-04-14 10:22, Daniel Thayer wrote: > On 4/14/17 10:46 AM, Azoff, Justin S wrote: >> >>> On Apr 14, 2017, at 10:34 AM, James Lay >>> wrote: >>> >>> and from crontab: >>> 0-59/5 * * * * /opt/bro/bin/broctl cron >>> >>> Cron job just isn't restarting these....manually running broctl cron >>> works though. >> >> Is the cron job definitely running? Is that in a users crontab or >> something in /etc/? If it's a system wide one in etc you'll need a >> user in there >> >> We use this: >> >> */5 * * * * root /bro/bin/broctl cron >> > > Another thing to check is to run this command: > broctl cron '?' > > The output should be: > cron enabled > > If it says "disabled", then "broctl cron" won't do anything. Yep...needed the user....sigh...I miss Slackware :( Thanks. James From finid at vivaldi.net Fri Apr 14 21:17:14 2017 From: finid at vivaldi.net (LinuxBSDos.com) Date: Fri, 14 Apr 2017 23:17:14 -0500 Subject: [Bro] Likely bug in broctl Message-ID: <3bd5b3cfd5eb50dbfde04bc9012f5ac7@vivaldi.net> Hello: On an Ubuntu 16.04 server, I have Bro executables in /usr/local/bro/bin, which is in the PATH of all users on the system. As root, I can invoke all the commands by just specifying the relative paths. As a standard user, I can do the same for the commands except broctl. If I type, for example, "broctl --h", it returns the following error messages: Traceback (most recent call last): File "/usr/local/bro/bin/broctl", line 830, in sys.exit(main()) File "/usr/local/bro/bin/broctl", line 797, in main loop = BroCtlCmdLoop(BroCtl, interactive, cmd) File "/usr/local/bro/bin/broctl", line 25, in __init__ self.broctl = broctl_class(ui=self) File "/usr/local/bro/lib/broctl/BroControl/broctl.py", line 69, in __init__ level=logging.DEBUG) File "/usr/lib/python2.7/logging/__init__.py", line 1547, in basicConfig hdlr = FileHandler(filename, mode) File "/usr/lib/python2.7/logging/__init__.py", line 913, in __init__ StreamHandler.__init__(self, self._open()) File "/usr/lib/python2.7/logging/__init__.py", line 943, in _open stream = open(self.baseFilename, self.mode) IOError: [Errno 13] Permission denied: '/usr/local/bro/spool/debug.log' "Permission denied" seems to make sense, but if I type "sudo broctl --h", I get "sudo: broctl: command not found". It only works if I specify the full path with sudo, so "sudo /usr/local/bro/bin/broctl --h" works, but "/usr/local/bro/bin/broctl --h" does not. Again, this is just for broctl. Thanks, -finid- Sent from my containerized Linux desktop ---------------------------------------- From dnthayer at illinois.edu Fri Apr 14 21:48:42 2017 From: dnthayer at illinois.edu (Daniel Thayer) Date: Fri, 14 Apr 2017 23:48:42 -0500 Subject: [Bro] Likely bug in broctl In-Reply-To: <3bd5b3cfd5eb50dbfde04bc9012f5ac7@vivaldi.net> References: <3bd5b3cfd5eb50dbfde04bc9012f5ac7@vivaldi.net> Message-ID: <6951eb67-3039-3a06-f1e7-f6ee6911418d@illinois.edu> If you upgrade to the newest release of Bro, then the error message is more user-friendly. The error is due to the fact that broctl needs to write to the "spool" and "logs" directories. This is explained in the broctl documentation: https://www.bro.org/sphinx/components/broctl/README.html#using-brocontrol-as-an-unprivileged-user On 4/14/17 11:17 PM, LinuxBSDos.com wrote: > Hello: > > On an Ubuntu 16.04 server, I have Bro executables in /usr/local/bro/bin, > which is in the PATH of all users on the system. > > As root, I can invoke all the commands by just specifying the relative > paths. As a standard user, I can do the same for the commands except > broctl. If I type, for example, "broctl --h", it returns the following > error messages: > > > Traceback (most recent call last): > File "/usr/local/bro/bin/broctl", line 830, in > sys.exit(main()) > File "/usr/local/bro/bin/broctl", line 797, in main > loop = BroCtlCmdLoop(BroCtl, interactive, cmd) > File "/usr/local/bro/bin/broctl", line 25, in __init__ > self.broctl = broctl_class(ui=self) > File "/usr/local/bro/lib/broctl/BroControl/broctl.py", line 69, in > __init__ > level=logging.DEBUG) > File "/usr/lib/python2.7/logging/__init__.py", line 1547, in > basicConfig > hdlr = FileHandler(filename, mode) > File "/usr/lib/python2.7/logging/__init__.py", line 913, in __init__ > StreamHandler.__init__(self, self._open()) > File "/usr/lib/python2.7/logging/__init__.py", line 943, in _open > stream = open(self.baseFilename, self.mode) > IOError: [Errno 13] Permission denied: '/usr/local/bro/spool/debug.log' > > > > "Permission denied" seems to make sense, but if I type "sudo broctl > --h", I get "sudo: broctl: command not found". > > It only works if I specify the full path with sudo, so "sudo > /usr/local/bro/bin/broctl --h" works, but "/usr/local/bro/bin/broctl > --h" does not. > > Again, this is just for broctl. > From jlay at slave-tothe-box.net Sat Apr 15 04:12:02 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Sat, 15 Apr 2017 05:12:02 -0600 Subject: [Bro] Likely bug in broctl In-Reply-To: <6951eb67-3039-3a06-f1e7-f6ee6911418d@illinois.edu> References: <3bd5b3cfd5eb50dbfde04bc9012f5ac7@vivaldi.net> <6951eb67-3039-3a06-f1e7-f6ee6911418d@illinois.edu> Message-ID: <1492254722.2529.25.camel@slave-tothe-box.net> On Fri, 2017-04-14 at 23:48 -0500, Daniel Thayer wrote: > If you upgrade to the newest release of Bro, then > the error message is more user-friendly. > > The error is due to the fact that broctl needs to > write to the "spool" and "logs" directories. > This is explained in the broctl documentation: > https://www.bro.org/sphinx/components/broctl/README.html#using-brocon > trol-as-an-unprivileged-user > > And as for /usr/local/bro/bin, add that to your secure_path= line in /etc/sudoers file. James? > On 4/14/17 11:17 PM, LinuxBSDos.com wrote: > > > > > Hello: > > > > On an Ubuntu 16.04 server, I have Bro executables in /usr/local/bro/bin, > > which is in the PATH of all users on the system. > > > > As root, I can invoke all the commands by just specifying the relative > > paths. As a standard user, I can do the same for the commands except > > broctl. If I type, for example, "broctl --h", it returns the following > > error messages: > > > > > > Traceback (most recent call last): > > File "/usr/local/bro/bin/broctl", line 830, in > > sys.exit(main()) > > File "/usr/local/bro/bin/broctl", line 797, in main > > loop = BroCtlCmdLoop(BroCtl, interactive, cmd) > > File "/usr/local/bro/bin/broctl", line 25, in __init__ > > self.broctl = broctl_class(ui=self) > > File "/usr/local/bro/lib/broctl/BroControl/broctl.py", line 69, in > > __init__ > > level=logging.DEBUG) > > File "/usr/lib/python2.7/logging/__init__.py", line 1547, in > > basicConfig > > hdlr = FileHandler(filename, mode) > > File "/usr/lib/python2.7/logging/__init__.py", line 913, in __init__ > > StreamHandler.__init__(self, self._open()) > > File "/usr/lib/python2.7/logging/__init__.py", line 943, in _open > > stream = open(self.baseFilename, self.mode) > > IOError: [Errno 13] Permission denied: '/usr/local/bro/spool/debug.log' > > > > > > > > "Permission denied" seems to make sense, but if I type "sudo broctl > > --h", I get "sudo: broctl: command not found". > > > > It only works if I specify the full path with sudo, so "sudo > > /usr/local/bro/bin/broctl --h" works, but "/usr/local/bro/bin/broctl > > --h" does not. > > > > Again, this is just for broctl. > > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170415/ee373f24/attachment.html From finid at vivaldi.net Sat Apr 15 09:22:08 2017 From: finid at vivaldi.net (LinuxBSDos.com) Date: Sat, 15 Apr 2017 11:22:08 -0500 Subject: [Bro] On Bro's configuration file Message-ID: <7e207a705d4e957925a60aa0d8f5b390@vivaldi.net> Hello: Just need some clarification on Bro's configuration files. 1. In node.cfg, what if I have two interfaces on a server that I'll like to monitor, can I add the second interface, like "interface=eth0,eth1"? 2. Regarding the networks.cfg file, it says it's a "List of local networks", while the docs says it's list of "networks that Bro will consider local to the monitored environment". By "local", does that mean _any_ IP address network associated with the server, including that that a private interface belongs to, and the loopback interface? Thanks, -- -finid- Sent from my containerized Linux desktop ---------------------------------------- From theomnipotentyouth at gmail.com Sun Apr 16 22:10:12 2017 From: theomnipotentyouth at gmail.com (RoM) Date: Mon, 17 Apr 2017 13:10:12 +0800 Subject: [Bro] Layer 7 DoS attacks Message-ID: Hi all, I saw an interesting post (http://mailman.icsi.berkeley. edu/pipermail/bro/2012-January/004508.html)about detecting layer 7 DoS attack using Bro, there was a script written by Seth Hall(seth at corelight.com )(http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/201201 09/84fdf6c0/attachment.obj ), but the script won't work in the new version of Bro, so I was wondering if anyone had any idea on how to do it in Bro 2.5? Thanks for any feedback in advance! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170417/bc74775c/attachment.html From pssunu6 at gmail.com Sun Apr 16 22:53:27 2017 From: pssunu6 at gmail.com (ps sunu) Date: Mon, 17 Apr 2017 11:23:27 +0530 Subject: [Bro] host_name adding into dhcp.log Message-ID: Hi, How to add host name in dhcp.log any samples there ? Regards, Sunu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170417/18f0b4b6/attachment.html From carlopmart at gmail.com Mon Apr 17 04:25:23 2017 From: carlopmart at gmail.com (C. L. Martinez) Date: Mon, 17 Apr 2017 11:25:23 +0000 Subject: [Bro] Ports used between manager/logger/proxy host and worker nodes Message-ID: <20170417112522.37mshye6yafbypki@scotland.uxdom.org> Hi all, I have setup one manager/logger/proxy host with 5 worker nodes (all using 2.5 version). Two of these 5 worker nodes are behind firewalls. I am seeing several packets dropped between these worker nodes and manager host: Apr 17 11:23:59.890910 rule 21/(match) [uid 0, pid 75183] block out on vio5: [uid 4294967295, pid 100000] 172.22.59.2.1255 > 172.22.59.4.47763: S 2230094890:2230094890(0) win 16384 (DF) (ttl 64, id 47383, len 64, bad ip cksum 14! -> b36d) Apr 17 11:23:59.890988 rule 21/(match) [uid 0, pid 75183] block out on vio5: [uid 4294967295, pid 100000] 172.22.59.2.35138 > 172.22.59.4.47762: S 4275416417:4275416417(0) win 16384 (DF) (ttl 64, id 42370, len 64, bad ip cksum 14! -> c702) Apr 17 11:23:59.891057 rule 21/(match) [uid 0, pid 75183] block out on vio5: [uid 4294967295, pid 100000] 172.22.59.2.24230 > 172.22.59.4.47761: S 363396747:363396747(0) win 16384 (DF) (ttl 64, id 38422, len 64, bad ip cksum 14! -> d66e) What ports do I need to open in these firewalls to permit comms between worker nodes and manager host? Thanks -- Greetings, C. L. Martinez From carlopmart at gmail.com Mon Apr 17 05:32:40 2017 From: carlopmart at gmail.com (C. L. Martinez) Date: Mon, 17 Apr 2017 12:32:40 +0000 Subject: [Bro] Ports used between manager/logger/proxy host and worker nodes In-Reply-To: <20170417112522.37mshye6yafbypki@scotland.uxdom.org> References: <20170417112522.37mshye6yafbypki@scotland.uxdom.org> Message-ID: <20170417123240.2ovba7nobdk2b4vk@scotland.uxdom.org> On Mon, Apr 17, 2017 at 11:25:23AM +0000, C. L. Martinez wrote: > Hi all, > > I have setup one manager/logger/proxy host with 5 worker nodes (all using 2.5 version). Two of these 5 worker nodes are behind firewalls. I am seeing several packets dropped between these worker nodes and manager host: > > Apr 17 11:23:59.890910 rule 21/(match) [uid 0, pid 75183] block out on vio5: [uid 4294967295, pid 100000] 172.22.59.2.1255 > 172.22.59.4.47763: S 2230094890:2230094890(0) win 16384 (DF) (ttl 64, id 47383, len 64, bad ip cksum 14! -> b36d) > Apr 17 11:23:59.890988 rule 21/(match) [uid 0, pid 75183] block out on vio5: [uid 4294967295, pid 100000] 172.22.59.2.35138 > 172.22.59.4.47762: S 4275416417:4275416417(0) win 16384 (DF) (ttl 64, id 42370, len 64, bad ip cksum 14! -> c702) > Apr 17 11:23:59.891057 rule 21/(match) [uid 0, pid 75183] block out on vio5: [uid 4294967295, pid 100000] 172.22.59.2.24230 > 172.22.59.4.47761: S 363396747:363396747(0) win 16384 (DF) (ttl 64, id 38422, len 64, bad ip cksum 14! -> d66e) > > What ports do I need to open in these firewalls to permit comms between worker nodes and manager host? > > Thanks > -- More info. According to broctl-config.sh, comms are established on port 47760: bindir="/opt/bro/bin" bro="/opt/bro/bin/bro" broargs="" brobase="/opt/bro" broctlconfigdir="/nsm/bro/spool" broport="47760" broscriptdir="/opt/bro/share/bro" capstatspath="/opt/bro/bin/capstats" cfgdir="/opt/bro/etc" .... But as you can see in previous log, worker nodes tries to connect to port 47763. Do I need to open a pool of ports on my firewalls? Can I configure what tcp port to use between workers and manager host? Thanks -- Greetings, C. L. Martinez From carlopmart at gmail.com Mon Apr 17 05:36:44 2017 From: carlopmart at gmail.com (C. L. Martinez) Date: Mon, 17 Apr 2017 12:36:44 +0000 Subject: [Bro] Ports used between manager/logger/proxy host and worker nodes (SOLVED) In-Reply-To: <20170417123240.2ovba7nobdk2b4vk@scotland.uxdom.org> References: <20170417112522.37mshye6yafbypki@scotland.uxdom.org> <20170417123240.2ovba7nobdk2b4vk@scotland.uxdom.org> Message-ID: <20170417123644.qyi7oembwiliw6ha@scotland.uxdom.org> On Mon, Apr 17, 2017 at 12:32:40PM +0000, C. L. Martinez wrote: > On Mon, Apr 17, 2017 at 11:25:23AM +0000, C. L. Martinez wrote: > > Hi all, > > > > I have setup one manager/logger/proxy host with 5 worker nodes (all using 2.5 version). Two of these 5 worker nodes are behind firewalls. I am seeing several packets dropped between these worker nodes and manager host: > > > > Apr 17 11:23:59.890910 rule 21/(match) [uid 0, pid 75183] block out on vio5: [uid 4294967295, pid 100000] 172.22.59.2.1255 > 172.22.59.4.47763: S 2230094890:2230094890(0) win 16384 (DF) (ttl 64, id 47383, len 64, bad ip cksum 14! -> b36d) > > Apr 17 11:23:59.890988 rule 21/(match) [uid 0, pid 75183] block out on vio5: [uid 4294967295, pid 100000] 172.22.59.2.35138 > 172.22.59.4.47762: S 4275416417:4275416417(0) win 16384 (DF) (ttl 64, id 42370, len 64, bad ip cksum 14! -> c702) > > Apr 17 11:23:59.891057 rule 21/(match) [uid 0, pid 75183] block out on vio5: [uid 4294967295, pid 100000] 172.22.59.2.24230 > 172.22.59.4.47761: S 363396747:363396747(0) win 16384 (DF) (ttl 64, id 38422, len 64, bad ip cksum 14! -> d66e) > > > > What ports do I need to open in these firewalls to permit comms between worker nodes and manager host? > > > > Thanks > > -- > > More info. According to broctl-config.sh, comms are established on port 47760: > > bindir="/opt/bro/bin" > bro="/opt/bro/bin/bro" > broargs="" > brobase="/opt/bro" > broctlconfigdir="/nsm/bro/spool" > broport="47760" > broscriptdir="/opt/bro/share/bro" > capstatspath="/opt/bro/bin/capstats" > cfgdir="/opt/bro/etc" > .... > > But as you can see in previous log, worker nodes tries to connect to port 47763. Do I need to open a pool of ports on my firewalls? Can I configure what tcp port to use between workers and manager host? > > Thanks > Ok, got it. According to Bro's manual: "Note that you can change the port that Bro listens on by changing the value of the ?BroPort? option in your broctl.cfg file (this should be needed only if your system has another process that listens on the same port). By default, a standalone Bro listens on TCP port 47760. For a cluster setup, the logger listens on TCP port 47761, and the manager listens on TCP port 47762 (or 47761 if no logger is defined). Each proxy is assigned its own port number, starting with one number greater than the manager?s port. Likewise, each worker is assigned its own port starting one number greater than the highest port number assigned to a proxy." Openning ports 47762 and 47761, it seems all works ok. -- Greetings, C. L. Martinez From seth at corelight.com Mon Apr 17 08:10:41 2017 From: seth at corelight.com (Seth Hall) Date: Mon, 17 Apr 2017 11:10:41 -0400 Subject: [Bro] Bro and GeoIP support In-Reply-To: References: Message-ID: <496B0224-8BEA-45B7-BE84-F2B60A58D38E@corelight.com> > On Apr 13, 2017, at 1:52 PM, LinuxBSDos.com wrote: > > If not, what's the process of getting Bro to support the mmdb database? I'm working on this and I'm going to be releasing it as a bro-pkg when it's functional. .Seth -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From seth at corelight.com Mon Apr 17 08:11:53 2017 From: seth at corelight.com (Seth Hall) Date: Mon, 17 Apr 2017 11:11:53 -0400 Subject: [Bro] Issue with Bro and libpcap during compile In-Reply-To: References: Message-ID: <9689341F-5963-424D-9D79-A3C175D75C17@corelight.com> > On Apr 13, 2017, at 3:38 PM, Dave Florek wrote: > > I'm curious to see if anyone in the Bro community has been successful at installing Bro from source using the initial libpcap files bundled with CentOS 7. No matter if I specify ./configure --with-pcap=/usr/local/ or ./configure --with-pcap=/usr/lib64/ after installing libpcap-devel, I'm still unable to get Bro to compile: I suppose you're using some libpcap wrapper since you're telling it to look for libpcap in a certain location? What libpcap wrapper are you using? .Seth -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From seth at corelight.com Mon Apr 17 09:13:16 2017 From: seth at corelight.com (Seth Hall) Date: Mon, 17 Apr 2017 12:13:16 -0400 Subject: [Bro] Specifying scripts for Bro to load In-Reply-To: References: Message-ID: <4CCE028B-6597-49E8-BD95-8897467B66B6@corelight.com> Huh, I'm not sure what that documentation is referring to. :) I just pushed some documentation updates for that page, but I'm not sure how often the development documentation is generated. You can see the change I committed in the repository though. Thanks, .Seth > On Apr 12, 2017, at 7:41 PM, LinuxBSDos.com wrote: > > Hi: > > On this documentation page [1], it says "all" can be used to specify > that Bro "perform all the default analysis that?s available". However > when I attach "all" to the Bro command, like "bro -i eth0 all" is exits > with an error: "fatal error: can't find all". > > I know that "local" is used to load site-specific scripts, what can I > use to load all available scripts, as indicated in the docs? > > [1] > https://www.bro.org/sphinx/quickstart/index.html#bro-as-a-command-line-utility > > Thanks, > > > -finid- > Sent from my containerized Linux desktop > ---------------------------------------- > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From dave.a.florek at gmail.com Mon Apr 17 09:24:12 2017 From: dave.a.florek at gmail.com (Dave Florek) Date: Mon, 17 Apr 2017 12:24:12 -0400 Subject: [Bro] Bro Digest, Vol 132, Issue 24 In-Reply-To: References: Message-ID: Hi Seth, I'm trying to get Bro to compile with the libpcap shared object initially installed with CentOS or from the development package: /usr/lib64/libpcap.so.1 /usr/local/lib/libpcap.so.1 On Mon, Apr 17, 2017 at 11:11 AM, wrote: > Send Bro mailing list submissions to > bro at bro.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > or, via email, send a message with subject or body 'help' to > bro-request at bro.org > > You can reach the person managing the list at > bro-owner at bro.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Bro digest..." > > > Today's Topics: > > 1. Layer 7 DoS attacks (RoM) > 2. host_name adding into dhcp.log (ps sunu) > 3. Ports used between manager/logger/proxy host and worker nodes > (C. L. Martinez) > 4. Re: Ports used between manager/logger/proxy host and worker > nodes (C. L. Martinez) > 5. Re: Ports used between manager/logger/proxy host and worker > nodes (SOLVED) (C. L. Martinez) > 6. Re: Bro and GeoIP support (Seth Hall) > 7. Re: Issue with Bro and libpcap during compile (Seth Hall) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 17 Apr 2017 13:10:12 +0800 > From: RoM > Subject: [Bro] Layer 7 DoS attacks > To: bro at bro.org > Message-ID: > mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi all, > > I saw an interesting post (http://mailman.icsi.berkeley. > edu/pipermail/bro/2012-January/004508.html)about detecting layer 7 DoS > attack using Bro, there was a script written by Seth Hall( > seth at corelight.com > )(http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/201201 > 09/84fdf6c0/attachment.obj > 20120109/84fdf6c0/attachment.obj>), > but the script won't work in the new version of Bro, so I was wondering if > anyone had any idea on how to do it in Bro 2.5? > > Thanks for any feedback in advance! > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/ > 20170417/bc74775c/attachment-0001.html > > ------------------------------ > > Message: 2 > Date: Mon, 17 Apr 2017 11:23:27 +0530 > From: ps sunu > Subject: [Bro] host_name adding into dhcp.log > To: bro at bro.org > Message-ID: > com> > Content-Type: text/plain; charset="utf-8" > > Hi, > How to add host name in dhcp.log any samples there ? > > Regards, > Sunu > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/ > 20170417/18f0b4b6/attachment-0001.html > > ------------------------------ > > Message: 3 > Date: Mon, 17 Apr 2017 11:25:23 +0000 > From: "C. L. Martinez" > Subject: [Bro] Ports used between manager/logger/proxy host and worker > nodes > To: bro at bro.org > Message-ID: <20170417112522.37mshye6yafbypki at scotland.uxdom.org> > Content-Type: text/plain; charset=utf-8 > > Hi all, > > I have setup one manager/logger/proxy host with 5 worker nodes (all using > 2.5 version). Two of these 5 worker nodes are behind firewalls. I am seeing > several packets dropped between these worker nodes and manager host: > > Apr 17 11:23:59.890910 rule 21/(match) [uid 0, pid 75183] block out on > vio5: [uid 4294967295, pid 100000] 172.22.59.2.1255 > 172.22.59.4.47763: S > 2230094890:2230094890(0) win 16384 6,nop,nop,timestamp 1780505936[|tcp]> (DF) (ttl 64, id 47383, len 64, bad > ip cksum 14! -> b36d) > Apr 17 11:23:59.890988 rule 21/(match) [uid 0, pid 75183] block out on > vio5: [uid 4294967295, pid 100000] 172.22.59.2.35138 > 172.22.59.4.47762: S > 4275416417:4275416417(0) win 16384 6,nop,nop,timestamp 1149589794[|tcp]> (DF) (ttl 64, id 42370, len 64, bad > ip cksum 14! -> c702) > Apr 17 11:23:59.891057 rule 21/(match) [uid 0, pid 75183] block out on > vio5: [uid 4294967295, pid 100000] 172.22.59.2.24230 > 172.22.59.4.47761: S > 363396747:363396747(0) win 16384 6,nop,nop,timestamp 703336159[|tcp]> (DF) (ttl 64, id 38422, len 64, bad ip > cksum 14! -> d66e) > > What ports do I need to open in these firewalls to permit comms between > worker nodes and manager host? > > Thanks > -- > Greetings, > C. L. Martinez > > > ------------------------------ > > Message: 4 > Date: Mon, 17 Apr 2017 12:32:40 +0000 > From: "C. L. Martinez" > Subject: Re: [Bro] Ports used between manager/logger/proxy host and > worker nodes > To: bro at bro.org > Message-ID: <20170417123240.2ovba7nobdk2b4vk at scotland.uxdom.org> > Content-Type: text/plain; charset=utf-8 > > On Mon, Apr 17, 2017 at 11:25:23AM +0000, C. L. Martinez wrote: > > Hi all, > > > > I have setup one manager/logger/proxy host with 5 worker nodes (all > using 2.5 version). Two of these 5 worker nodes are behind firewalls. I am > seeing several packets dropped between these worker nodes and manager host: > > > > Apr 17 11:23:59.890910 rule 21/(match) [uid 0, pid 75183] block out on > vio5: [uid 4294967295, pid 100000] 172.22.59.2.1255 > 172.22.59.4.47763: S > 2230094890:2230094890(0) win 16384 6,nop,nop,timestamp 1780505936[|tcp]> (DF) (ttl 64, id 47383, len 64, bad > ip cksum 14! -> b36d) > > Apr 17 11:23:59.890988 rule 21/(match) [uid 0, pid 75183] block out on > vio5: [uid 4294967295, pid 100000] 172.22.59.2.35138 > 172.22.59.4.47762: S > 4275416417:4275416417(0) win 16384 6,nop,nop,timestamp 1149589794[|tcp]> (DF) (ttl 64, id 42370, len 64, bad > ip cksum 14! -> c702) > > Apr 17 11:23:59.891057 rule 21/(match) [uid 0, pid 75183] block out on > vio5: [uid 4294967295, pid 100000] 172.22.59.2.24230 > 172.22.59.4.47761: S > 363396747:363396747(0) win 16384 6,nop,nop,timestamp 703336159[|tcp]> (DF) (ttl 64, id 38422, len 64, bad ip > cksum 14! -> d66e) > > > > What ports do I need to open in these firewalls to permit comms between > worker nodes and manager host? > > > > Thanks > > -- > > More info. According to broctl-config.sh, comms are established on port > 47760: > > bindir="/opt/bro/bin" > bro="/opt/bro/bin/bro" > broargs="" > brobase="/opt/bro" > broctlconfigdir="/nsm/bro/spool" > broport="47760" > broscriptdir="/opt/bro/share/bro" > capstatspath="/opt/bro/bin/capstats" > cfgdir="/opt/bro/etc" > .... > > But as you can see in previous log, worker nodes tries to connect to port > 47763. Do I need to open a pool of ports on my firewalls? Can I configure > what tcp port to use between workers and manager host? > > Thanks > > -- > Greetings, > C. L. Martinez > > > ------------------------------ > > Message: 5 > Date: Mon, 17 Apr 2017 12:36:44 +0000 > From: "C. L. Martinez" > Subject: Re: [Bro] Ports used between manager/logger/proxy host and > worker nodes (SOLVED) > To: bro at bro.org > Message-ID: <20170417123644.qyi7oembwiliw6ha at scotland.uxdom.org> > Content-Type: text/plain; charset=utf-8 > > On Mon, Apr 17, 2017 at 12:32:40PM +0000, C. L. Martinez wrote: > > On Mon, Apr 17, 2017 at 11:25:23AM +0000, C. L. Martinez wrote: > > > Hi all, > > > > > > I have setup one manager/logger/proxy host with 5 worker nodes (all > using 2.5 version). Two of these 5 worker nodes are behind firewalls. I am > seeing several packets dropped between these worker nodes and manager host: > > > > > > Apr 17 11:23:59.890910 rule 21/(match) [uid 0, pid 75183] block out on > vio5: [uid 4294967295, pid 100000] 172.22.59.2.1255 > 172.22.59.4.47763: S > 2230094890:2230094890(0) win 16384 6,nop,nop,timestamp 1780505936[|tcp]> (DF) (ttl 64, id 47383, len 64, bad > ip cksum 14! -> b36d) > > > Apr 17 11:23:59.890988 rule 21/(match) [uid 0, pid 75183] block out on > vio5: [uid 4294967295, pid 100000] 172.22.59.2.35138 > 172.22.59.4.47762: S > 4275416417:4275416417(0) win 16384 6,nop,nop,timestamp 1149589794[|tcp]> (DF) (ttl 64, id 42370, len 64, bad > ip cksum 14! -> c702) > > > Apr 17 11:23:59.891057 rule 21/(match) [uid 0, pid 75183] block out on > vio5: [uid 4294967295, pid 100000] 172.22.59.2.24230 > 172.22.59.4.47761: S > 363396747:363396747(0) win 16384 6,nop,nop,timestamp 703336159[|tcp]> (DF) (ttl 64, id 38422, len 64, bad ip > cksum 14! -> d66e) > > > > > > What ports do I need to open in these firewalls to permit comms > between worker nodes and manager host? > > > > > > Thanks > > > -- > > > > More info. According to broctl-config.sh, comms are established on port > 47760: > > > > bindir="/opt/bro/bin" > > bro="/opt/bro/bin/bro" > > broargs="" > > brobase="/opt/bro" > > broctlconfigdir="/nsm/bro/spool" > > broport="47760" > > broscriptdir="/opt/bro/share/bro" > > capstatspath="/opt/bro/bin/capstats" > > cfgdir="/opt/bro/etc" > > .... > > > > But as you can see in previous log, worker nodes tries to connect to > port 47763. Do I need to open a pool of ports on my firewalls? Can I > configure what tcp port to use between workers and manager host? > > > > Thanks > > > > Ok, got it. According to Bro's manual: > > "Note that you can change the port that Bro listens on by changing the > value of the ?BroPort? option in your broctl.cfg file (this should be > needed only if your system has another process that listens on the same > port). By default, a standalone Bro listens on TCP port 47760. For a > cluster setup, the logger listens on TCP port 47761, and the manager > listens on TCP port 47762 (or 47761 if no logger is defined). Each proxy is > assigned its own port number, starting with one number greater than the > manager?s port. Likewise, each worker is assigned its own port starting one > number greater than the highest port number assigned to a proxy." > > Openning ports 47762 and 47761, it seems all works ok. > > -- > Greetings, > C. L. Martinez > > > ------------------------------ > > Message: 6 > Date: Mon, 17 Apr 2017 11:10:41 -0400 > From: Seth Hall > Subject: Re: [Bro] Bro and GeoIP support > To: "LinuxBSDos.com" > Cc: Bro > Message-ID: <496B0224-8BEA-45B7-BE84-F2B60A58D38E at corelight.com> > Content-Type: text/plain; charset=us-ascii > > > > On Apr 13, 2017, at 1:52 PM, LinuxBSDos.com wrote: > > > > If not, what's the process of getting Bro to support the mmdb database? > > I'm working on this and I'm going to be releasing it as a bro-pkg when > it's functional. > > .Seth > > -- > Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com > > > > > ------------------------------ > > Message: 7 > Date: Mon, 17 Apr 2017 11:11:53 -0400 > From: Seth Hall > Subject: Re: [Bro] Issue with Bro and libpcap during compile > To: Dave Florek > Cc: bro at bro.org > Message-ID: <9689341F-5963-424D-9D79-A3C175D75C17 at corelight.com> > Content-Type: text/plain; charset=us-ascii > > > > On Apr 13, 2017, at 3:38 PM, Dave Florek > wrote: > > > > I'm curious to see if anyone in the Bro community has been successful at > installing Bro from source using the initial libpcap files bundled with > CentOS 7. No matter if I specify ./configure --with-pcap=/usr/local/ or > ./configure --with-pcap=/usr/lib64/ after installing libpcap-devel, I'm > still unable to get Bro to compile: > > I suppose you're using some libpcap wrapper since you're telling it to > look for libpcap in a certain location? What libpcap wrapper are you using? > > .Seth > > -- > Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com > > > > > ------------------------------ > > _______________________________________________ > Bro mailing list > Bro at bro.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > End of Bro Digest, Vol 132, Issue 24 > ************************************ > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170417/4dd4f8c5/attachment-0001.html From seth at corelight.com Mon Apr 17 10:24:48 2017 From: seth at corelight.com (Seth Hall) Date: Mon, 17 Apr 2017 13:24:48 -0400 Subject: [Bro] Layer 7 DoS attacks In-Reply-To: References: Message-ID: D'oh! Sorry, I ran a version I have on my laptop which was updated. I attached the version of the script that works now... -------------- next part -------------- A non-text attachment was scrubbed... Name: http-DoS-detector.bro Type: application/octet-stream Size: 2478 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170417/555003c8/attachment.obj -------------- next part -------------- .Seth > On Apr 17, 2017, at 1:10 AM, RoM wrote: > > Hi all, > > I saw an interesting post (http://mailman.icsi.berkeley.edu/pipermail/bro/2012-January/004508.html)about detecting layer 7 DoS attack using Bro, there was a script written by Seth Hall(seth at corelight.com)(http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120109/84fdf6c0/attachment.obj), but the script won't work in the new version of Bro, so I was wondering if anyone had any idea on how to do it in Bro 2.5? > > Thanks for any feedback in advance! > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From seth at corelight.com Mon Apr 17 10:26:09 2017 From: seth at corelight.com (Seth Hall) Date: Mon, 17 Apr 2017 13:26:09 -0400 Subject: [Bro] host_name adding into dhcp.log In-Reply-To: References: Message-ID: > On Apr 17, 2017, at 1:53 AM, ps sunu wrote: > > How to add host name in dhcp.log any samples there ? Unfortunately this isn't terribly straightforward due to how DHCP works (broadcast requests and unicast replies). I am working on a local branch that adds this feature and makes it possible to extend the DHCP log with other information too. This should make it into 2.6 .Seth -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From seth at corelight.com Mon Apr 17 10:28:03 2017 From: seth at corelight.com (Seth Hall) Date: Mon, 17 Apr 2017 13:28:03 -0400 Subject: [Bro] Bro Digest, Vol 132, Issue 24 In-Reply-To: References: Message-ID: <7F75D63E-F867-4BF7-9C3B-160C838021ED@corelight.com> > On Apr 17, 2017, at 12:24 PM, Dave Florek wrote: > > I'm trying to get Bro to compile with the libpcap shared object initially installed with CentOS or from the development package: > > /usr/lib64/libpcap.so.1 > /usr/local/lib/libpcap.so.1 What package did /usr/local/lib/libpcap.so.1 come from? CentOS shouldn't install the "normal" libpcap library there. .Seth -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From bill.de.ping at gmail.com Mon Apr 17 23:54:35 2017 From: bill.de.ping at gmail.com (william de ping) Date: Tue, 18 Apr 2017 09:54:35 +0300 Subject: [Bro] define a variable name based on string in bro Message-ID: Hi all, I am trying to auto generate a variable name dynamically. For example, I want to create a test_i variable were i is a digit, that var will be used as the name of a set : event bro_init() { for (i in set(1,3,5,7)) { local fmt("test_%s",i): set[string] = {"one","two","three"}; } } event bro_done() { for (i in set(1,3,5,7)) { print "one" in fmt("test_%s",i); } } The above does not work, so is there any command in bro language that could transform a string into a variable ? Thanks B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170418/2aaa650a/attachment.html From jan.grashoefer at gmail.com Tue Apr 18 02:27:52 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Tue, 18 Apr 2017 11:27:52 +0200 Subject: [Bro] define a variable name based on string in bro In-Reply-To: References: Message-ID: <73f3b8e0-b2a2-7522-edf2-907a543b5c47@gmail.com> > I am trying to auto generate a variable name dynamically. > ... > The above does not work, so is there any command in bro language that could > transform a string into a variable ? Seems what you're looking for is a string indexed table: https://www.bro.org/sphinx-git/scripting/index.html#id12 Jan From bill.de.ping at gmail.com Tue Apr 18 04:07:13 2017 From: bill.de.ping at gmail.com (william de ping) Date: Tue, 18 Apr 2017 14:07:13 +0300 Subject: [Bro] define a variable name based on string in bro In-Reply-To: <73f3b8e0-b2a2-7522-edf2-907a543b5c47@gmail.com> References: <73f3b8e0-b2a2-7522-edf2-907a543b5c47@gmail.com> Message-ID: Hi Jan, Thank you, but I fail to see what it is your speaking of in the link ? Do you mean that each element in the table is a variable ? In that case, writing\reading from this table will have complexity of O(n), right ? Thanks again B On Tue, Apr 18, 2017 at 12:27 PM, Jan Grash?fer wrote: > > I am trying to auto generate a variable name dynamically. > > ... > > The above does not work, so is there any command in bro language that > could > > transform a string into a variable ? > > Seems what you're looking for is a string indexed table: > https://www.bro.org/sphinx-git/scripting/index.html#id12 > > Jan > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170418/ec6c87f1/attachment.html From jan.grashoefer at gmail.com Tue Apr 18 04:15:01 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Tue, 18 Apr 2017 13:15:01 +0200 Subject: [Bro] define a variable name based on string in bro In-Reply-To: References: <73f3b8e0-b2a2-7522-edf2-907a543b5c47@gmail.com> Message-ID: <522fd09f-66f5-eec5-28dc-d6931337e5a7@gmail.com> > Thank you, but I fail to see what it is your speaking of in the link ? > Do you mean that each element in the table is a variable ? Exactly. > In that case, writing\reading from this table will have complexity of O(n), > right ? No, it should be O(1) on average, see https://www.bro.org/sphinx/script-reference/types.html#type-table. I hope this helps, Jan From bill.de.ping at gmail.com Tue Apr 18 04:16:43 2017 From: bill.de.ping at gmail.com (william de ping) Date: Tue, 18 Apr 2017 14:16:43 +0300 Subject: [Bro] - set\table\vector types have a complexity of O(n) ? Message-ID: Hi all, Just wondering, sets\tables\vectors all have a read\write complexity of O(n) ? n - referring to the number of elements in the container. Thanks B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170418/1f466654/attachment.html From bill.de.ping at gmail.com Tue Apr 18 04:23:05 2017 From: bill.de.ping at gmail.com (william de ping) Date: Tue, 18 Apr 2017 14:23:05 +0300 Subject: [Bro] define a variable name based on string in bro In-Reply-To: <522fd09f-66f5-eec5-28dc-d6931337e5a7@gmail.com> References: <73f3b8e0-b2a2-7522-edf2-907a543b5c47@gmail.com> <522fd09f-66f5-eec5-28dc-d6931337e5a7@gmail.com> Message-ID: Hi Jan, I wonder if O(1) is the case for reading from a table as well ? If so, what would be the benefit of using a bloom filter ? Thanks B On Tue, Apr 18, 2017 at 2:15 PM, Jan Grash?fer wrote: > > Thank you, but I fail to see what it is your speaking of in the link ? > > Do you mean that each element in the table is a variable ? > > Exactly. > > > In that case, writing\reading from this table will have complexity of > O(n), > > right ? > > No, it should be O(1) on average, see > https://www.bro.org/sphinx/script-reference/types.html#type-table. > > I hope this helps, > Jan > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170418/049f807d/attachment.html From jan.grashoefer at gmail.com Tue Apr 18 04:36:43 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Tue, 18 Apr 2017 13:36:43 +0200 Subject: [Bro] define a variable name based on string in bro In-Reply-To: References: <73f3b8e0-b2a2-7522-edf2-907a543b5c47@gmail.com> <522fd09f-66f5-eec5-28dc-d6931337e5a7@gmail.com> Message-ID: <830d6283-a2b9-d99e-39fb-7187b3486e39@gmail.com> > I wonder if O(1) is the case for reading from a table as well ? > If so, what would be the benefit of using a bloom filter ? Roughly said: You cannot store the actual data in a bloom filter. It is a probabilistic data structure for membership testing only. Compared to hash tables, bloom filters are very small and constant in space. Jan From jan.grashoefer at gmail.com Tue Apr 18 04:49:08 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Tue, 18 Apr 2017 13:49:08 +0200 Subject: [Bro] - set\table\vector types have a complexity of O(n) ? In-Reply-To: References: Message-ID: <0388a196-23bc-17fb-3382-a8b6a2c12779@gmail.com> > Just wondering, sets\tables\vectors all have a read\write complexity of > O(n) ? > n - referring to the number of elements in the container. If I am not mistaken, sets as well as tables are implemented as hash tables. Thus the average complexity for lookup and insert is O(1). Vectors are implemented using C++ vectors, I think. I.e., lookup would be O(1) while inserting depends on the context. Jan From bill.de.ping at gmail.com Tue Apr 18 04:57:40 2017 From: bill.de.ping at gmail.com (william de ping) Date: Tue, 18 Apr 2017 14:57:40 +0300 Subject: [Bro] define a variable name based on string in bro In-Reply-To: <830d6283-a2b9-d99e-39fb-7187b3486e39@gmail.com> References: <73f3b8e0-b2a2-7522-edf2-907a543b5c47@gmail.com> <522fd09f-66f5-eec5-28dc-d6931337e5a7@gmail.com> <830d6283-a2b9-d99e-39fb-7187b3486e39@gmail.com> Message-ID: So testing if an element was seen before using a bloom filter has the same complexity as checking if its in a table ? Thanks B On Tue, Apr 18, 2017 at 2:36 PM, Jan Grash?fer wrote: > > I wonder if O(1) is the case for reading from a table as well ? > > If so, what would be the benefit of using a bloom filter ? > > Roughly said: You cannot store the actual data in a bloom filter. It is > a probabilistic data structure for membership testing only. Compared to > hash tables, bloom filters are very small and constant in space. > > Jan > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170418/8f48a017/attachment.html From jan.grashoefer at gmail.com Tue Apr 18 05:06:23 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Tue, 18 Apr 2017 14:06:23 +0200 Subject: [Bro] define a variable name based on string in bro In-Reply-To: References: <73f3b8e0-b2a2-7522-edf2-907a543b5c47@gmail.com> <522fd09f-66f5-eec5-28dc-d6931337e5a7@gmail.com> <830d6283-a2b9-d99e-39fb-7187b3486e39@gmail.com> Message-ID: <076f15c5-069e-5182-9eba-a44e055ce4bd@gmail.com> > So testing if an element was seen before using a bloom filter has the same > complexity as checking if its in a table ? It is a different tradeoff: The worst case for a hash table would be a lookup that takes O(n). The "worst case" for a bloom filter would be a match although the tested element was not added to the filter. Have a look at https://en.wikipedia.org/wiki/Hash_table and https://en.wikipedia.org/wiki/Bloom_filter for more details about these data structures. Jan From George.Macon at gtri.gatech.edu Tue Apr 18 11:38:08 2017 From: George.Macon at gtri.gatech.edu (George Macon) Date: Tue, 18 Apr 2017 14:38:08 -0400 Subject: [Bro] PySubnetTree on PyPI Message-ID: <4f3208fb-5947-7da2-c39d-f10c72062208@gtri.gatech.edu> The PySubnetTree library was uploaded to PyPI in 2014 when it was on version 0.23; this is still the most recent version on PyPI. This had been originally requested in the GitHub Issue #1, which I note was never closed. I asked on IRC where the appropriate place to ask about getting the most recent version uploaded and was directed to the mailing list. Can whoever controls the "bro" account on PyPI upload the newest version of PySubnetTree? Thanks, George From vern at berkeley.edu Tue Apr 18 15:17:42 2017 From: vern at berkeley.edu (Vern Paxson) Date: Tue, 18 Apr 2017 15:17:42 -0700 Subject: [Bro] define a variable name based on string in bro In-Reply-To: <076f15c5-069e-5182-9eba-a44e055ce4bd@gmail.com> (Tue, 18 Apr 2017 14:06:23 +0200). Message-ID: <20170418221742.23C162C4070@rock.ICSI.Berkeley.EDU> > It is a different tradeoff: The worst case for a hash table would be a > lookup that takes O(n). While that's indeed the worst case, note that Bro uses randomized universal hash functions, so in practice you should never see such a run-time, even for maliciously crafted inputs. Vern From pssunu6 at gmail.com Wed Apr 19 05:13:12 2017 From: pssunu6 at gmail.com (ps sunu) Date: Wed, 19 Apr 2017 17:43:12 +0530 Subject: [Bro] detect-tor.gro Message-ID: Hi, from below detect-tor script i need ssl session id, inside notice log @load base/frameworks/notice module DetectTor; export { redef enum Notice::Type += { ## Indicates that a host using Tor was discovered. DetectTor::Found }; ## Distinct Tor-like X.509 certificates to see before deciding it's Tor. const tor_cert_threshold = 1.0; ## Time period to see the :bro:see:`tor_cert_threshold` certificates ## before deciding it's Tor. const tor_cert_period = 5min; # Number of Tor certificate samples to collect. const tor_cert_samples = 3 &redef; } event bro_init() { local r1 = SumStats::Reducer($stream="ssl.tor-looking-cert", $apply=set(SumStats::UNIQUE, SumStats::SAMPLE), $num_samples=tor_cert_samples); SumStats::create([$name="detect-tor", $epoch=tor_cert_period, $reducers=set(r1), $threshold_val(key: SumStats::Key, result: SumStats::Result) = { return result["ssl.tor-looking-cert"]$unique+0.0; }, $threshold=tor_cert_threshold, $threshold_crossed(key: SumStats::Key, result: SumStats::Result) = { local r = result["ssl.tor-looking-cert"]; local samples = r$samples; local sub_msg = fmt("Sampled certificates: "); for ( i in samples ) { if ( samples[i]?$str ) sub_msg = fmt("%s%s %s", sub_msg, i==0 ? "":",", samples[i]$str); print r; } NOTICE([$note=DetectTor::Found, $msg=fmt("%s was found using Tor by connecting to servers with at least %d unique weird certs", key$host, r$unique), $id= ?? $session_id=?? $sub=sub_msg, $src=key$host, $identifier=cat(key$host)]); }]); } event ssl_established(c: connection ) { if ( c$ssl?$subject && /^CN=www.[^=,]*$/ == c$ssl$subject && c$ssl?$issuer && /^CN=www.[^=,]*$/ == c$ssl$issuer ) { SumStats::observe("ssl.tor-looking-cert", [$host=c$id$orig_h], [$str=c$ssl$subject]); } #print c$ssl; } Regards, Sunu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170419/dcba8382/attachment.html From pssunu6 at gmail.com Wed Apr 19 05:14:29 2017 From: pssunu6 at gmail.com (ps sunu) Date: Wed, 19 Apr 2017 17:44:29 +0530 Subject: [Bro] Fwd: detect-tor.gro In-Reply-To: References: Message-ID: Hi, from below detect-tor script i need ssl session id, and connection id (orig_h,p etc), inside notice log @load base/frameworks/notice module DetectTor; export { redef enum Notice::Type += { ## Indicates that a host using Tor was discovered. DetectTor::Found }; ## Distinct Tor-like X.509 certificates to see before deciding it's Tor. const tor_cert_threshold = 1.0; ## Time period to see the :bro:see:`tor_cert_threshold` certificates ## before deciding it's Tor. const tor_cert_period = 5min; # Number of Tor certificate samples to collect. const tor_cert_samples = 3 &redef; } event bro_init() { local r1 = SumStats::Reducer($stream="ssl.tor-looking-cert", $apply=set(SumStats::UNIQUE, SumStats::SAMPLE), $num_samples=tor_cert_samples); SumStats::create([$name="detect-tor", $epoch=tor_cert_period, $reducers=set(r1), $threshold_val(key: SumStats::Key, result: SumStats::Result) = { return result["ssl.tor-looking-cert"]$unique+0.0; }, $threshold=tor_cert_threshold, $threshold_crossed(key: SumStats::Key, result: SumStats::Result) = { local r = result["ssl.tor-looking-cert"]; local samples = r$samples; local sub_msg = fmt("Sampled certificates: "); for ( i in samples ) { if ( samples[i]?$str ) sub_msg = fmt("%s%s %s", sub_msg, i==0 ? "":",", samples[i]$str); print r; } NOTICE([$note=DetectTor::Found, $msg=fmt("%s was found using Tor by connecting to servers with at least %d unique weird certs", key$host, r$unique), $id= ?? $session_id=?? $sub=sub_msg, $src=key$host, $identifier=cat(key$host)]); }]); } event ssl_established(c: connection ) { if ( c$ssl?$subject && /^CN=www.[^=,]*$/ == c$ssl$subject && c$ssl?$issuer && /^CN=www.[^=,]*$/ == c$ssl$issuer ) { SumStats::observe("ssl.tor-looking-cert", [$host=c$id$orig_h], [$str=c$ssl$subject]); } #print c$ssl; } Regards, Sunu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170419/1994cff6/attachment-0001.html From bro at pingtrip.com Wed Apr 19 11:24:17 2017 From: bro at pingtrip.com (Dave Crawford) Date: Wed, 19 Apr 2017 14:24:17 -0400 Subject: [Bro] Combining Vectors Message-ID: <9236AEAD-A85B-4D73-817A-28F10F6103DC@pingtrip.com> Is there a built-in for combining two vectors, or is a for loop the approach? As an example: vec1 = vector( 1, 2, 3 ); vec2 = vector( 4, 5, 6 ); ...Magic happens here... vec3 = (1,2,3,4,5,6) -Dave From dave.a.florek at gmail.com Wed Apr 19 11:34:58 2017 From: dave.a.florek at gmail.com (Dave Florek) Date: Wed, 19 Apr 2017 14:34:58 -0400 Subject: [Bro] Bro Digest, Vol 132, Issue 24 In-Reply-To: <7F75D63E-F867-4BF7-9C3B-160C838021ED@corelight.com> References: <7F75D63E-F867-4BF7-9C3B-160C838021ED@corelight.com> Message-ID: Hi Seth, Apologies for the delay. It came from a PF_RING source package. /usr/lib64/libpcap.so.1 /usr/lib64/libpcap.so.1.5.3 /usr/local/lib/libpcap.so.1 /usr/local/lib/libpcap.so.1.7.4 ~/PF_RING/userland/libpcap-1.7.4/libpcap.so.1.7.4 I was able to resolve the issue by uninstalling the package and deleting the shared object. Thanks, On Mon, Apr 17, 2017 at 1:28 PM, Seth Hall wrote: > > > On Apr 17, 2017, at 12:24 PM, Dave Florek > wrote: > > > > I'm trying to get Bro to compile with the libpcap shared object > initially installed with CentOS or from the development package: > > > > /usr/lib64/libpcap.so.1 > > /usr/local/lib/libpcap.so.1 > > What package did /usr/local/lib/libpcap.so.1 come from? CentOS shouldn't > install the "normal" libpcap library there. > > .Seth > > -- > Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170419/152d6ae9/attachment.html From hovsep.sanjay.levi at gmail.com Wed Apr 19 11:44:35 2017 From: hovsep.sanjay.levi at gmail.com (Hovsep Levi) Date: Wed, 19 Apr 2017 18:44:35 +0000 Subject: [Bro] Patch for multiple loggers Message-ID: Finally made this work, the previous changes didn't assign a logger to the manager and proxies. Patches attached modify - lib/broctl/BroControl/install.py - lib/broctl/BroControl/config.py To use, adjust node.cfg to include logger-n entries, similar to proxies. Memory usage remains stable over time.... (so far) Wed Apr 19 18:06:21 UTC 2017 Checking Bro status... Name Type Host Pid Proc VSize Rss Cpu Cmd logger-1 logger 10.1.1.16 24527 parent 744M 225M 56% bro logger-1 logger 10.1.1.16 25476 child 174M 87M 4% bro logger-10 logger 10.1.1.16 24540 parent 731M 239M 51% bro logger-10 logger 10.1.1.16 25087 child 154M 94M 3% bro logger-11 logger 10.1.1.16 24543 parent 723M 222M 54% bro logger-11 logger 10.1.1.16 25390 child 154M 94M 3% bro logger-12 logger 10.1.1.16 24559 parent 719M 230M 54% bro logger-12 logger 10.1.1.16 25197 child 138M 77M 3% bro logger-2 logger 10.1.1.16 24557 parent 719M 228M 53% bro logger-2 logger 10.1.1.16 25477 child 154M 92M 3% bro logger-3 logger 10.1.1.16 24577 parent 715M 229M 55% bro logger-3 logger 10.1.1.16 25086 child 150M 90M 3% bro logger-4 logger 10.1.1.16 24585 parent 723M 234M 53% bro logger-4 logger 10.1.1.16 25204 child 138M 78M 3% bro logger-5 logger 10.1.1.16 24587 parent 727M 224M 54% bro logger-5 logger 10.1.1.16 25499 child 162M 97M 3% bro logger-6 logger 10.1.1.16 24593 parent 711M 229M 57% bro logger-6 logger 10.1.1.16 25366 child 142M 83M 3% bro logger-7 logger 10.1.1.16 24599 parent 715M 229M 53% bro logger-7 logger 10.1.1.16 25480 child 154M 95M 3% bro logger-8 logger 10.1.1.16 24600 parent 747M 239M 54% bro logger-8 logger 10.1.1.16 25166 child 142M 82M 3% bro logger-9 logger 10.1.1.16 24606 parent 723M 218M 60% bro logger-9 logger 10.1.1.16 25481 child 150M 91M 3% bro manager manager 10.1.1.16 25449 child 522M 256M 100% bro manager manager 10.1.1.16 25303 parent 566M 506M 27% bro Loggers using more CPU.... last pid: 36661; load averages: 20.99, 26.13, 75.80757 up 3+04:59:57 18:10:51 89 processes: 3 running, 86 sleeping CPU: 21.6% user, 0.6% nice, 15.1% system, 0.6% interrupt, 62.1% idle Mem: 1920M Active, 3494M Inact, 19G Wired, 35M Cache, 100G Free ARC: 7603M Total, 2708M MFU, 4469M MRU, 16K Anon, 50M Header, 377M Other Swap: 12G Total, 17M Used, 12G Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 25449 bro 1 108 5 522M 256M CPU21 21 22:53 100.00% bro 24593 bro 157 20 0 711M 229M select 27 42:15 72.46% bro 24587 bro 162 20 0 727M 224M select 26 41:58 71.58% bro 24557 bro 157 20 0 719M 228M select 21 42:07 70.70% bro 24606 bro 162 20 0 723M 218M select 31 42:12 70.61% bro 24600 bro 162 20 0 747M 239M select 17 42:11 70.51% bro 24540 bro 157 20 0 731M 239M select 6 41:33 70.46% bro 24585 bro 157 20 0 723M 235M select 21 41:48 69.53% bro 24543 bro 162 20 0 723M 222M select 7 42:05 68.75% bro 24577 bro 157 20 0 715M 229M select 34 42:03 67.72% bro 24599 bro 157 20 0 715M 229M select 21 41:08 64.60% bro 24527 bro 167 20 0 744M 226M select 19 43:20 64.11% bro 24559 bro 157 20 0 719M 231M select 36 42:05 62.50% bro 25303 bro 7 20 0 574M 512M uwait 43 7:35 27.98% bro 36661 bro 1 79 0 112M 19248K CPU19 19 0:03 23.39% python2.7 36449 bro 1 52 0 52696K 7992K select 10 0:26 19.29% ssh 36451 bro 1 52 0 52696K 7992K select 25 0:26 19.29% ssh 36450 bro 1 52 0 52696K 7992K select 36 0:26 18.99% ssh 36452 bro 1 52 0 17100K 2404K piperd 41 0:26 18.46% sh 25476 bro 1 28 5 174M 89224K select 19 2:01 5.47% bro 25166 bro 1 27 5 142M 84528K select 2 1:40 4.69% bro 25499 bro 1 27 5 162M 99636K select 39 1:38 4.49% bro 25366 bro 1 27 5 142M 85204K select 3 1:41 4.39% bro 25481 bro 1 27 5 150M 93376K select 0 1:40 4.30% bro 25480 bro 1 27 5 154M 97280K select 7 1:40 4.30% bro 25087 bro 1 27 5 154M 96464K select 16 1:38 4.30% bro 25390 bro 1 27 5 154M 97024K select 9 1:39 4.20% bro 25086 bro 1 27 5 150M 92540K select 26 1:37 4.20% bro 25477 bro 1 27 5 154M 94392K select 34 1:39 4.05% bro 25197 bro 1 27 5 138M 79808K select 24 1:40 3.96% bro 25204 bro 1 27 5 138M 80316K select 43 1:35 3.96% bro 28300 bro 1 20 0 21952K 3204K CPU16 16 0:26 1.27% top Mostly even distribution of packets across workers.. tcpdump -tnn -c 2000 -i lagg1 src portrange 47761-47780 | awk -F "." '{print $1"."$2"."$3"."$4"--"$5}' | sort | uniq -c | sort -nr | awk '{print $1, $2, $3}' | sort -k3 2000 packets captured 16263 packets received by filter 0 packets dropped by kernel 113 IP 10.1.1.16--47761 132 IP 10.1.1.16--47762 138 IP 10.1.1.16--47763 114 IP 10.1.1.16--47764 105 IP 10.1.1.16--47765 118 IP 10.1.1.16--47766 99 IP 10.1.1.16--47767 115 IP 10.1.1.16--47768 120 IP 10.1.1.16--47769 105 IP 10.1.1.16--47770 105 IP 10.1.1.16--47771 105 IP 10.1.1.16--47772 631 IP 10.1.1.16--47773 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170419/9bbe899d/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: multi-logger__config.py.patch Type: text/x-patch Size: 670 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170419/9bbe899d/attachment-0002.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: multi-logger__install.py.patch Type: text/x-patch Size: 3954 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170419/9bbe899d/attachment-0003.bin From bro at pingtrip.com Thu Apr 20 10:32:34 2017 From: bro at pingtrip.com (Dave Crawford) Date: Thu, 20 Apr 2017 13:32:34 -0400 Subject: [Bro] Combining Vectors In-Reply-To: <9236AEAD-A85B-4D73-817A-28F10F6103DC@pingtrip.com> References: <9236AEAD-A85B-4D73-817A-28F10F6103DC@pingtrip.com> Message-ID: <48FBC4B2-8EE7-4253-8B42-535038F35B73@pingtrip.com> This is how I implemented the ?merge? I needed: for (i in vec1) { vec3[|vec3|] = vec1[i]; } for (i in vec2) { vec3[|vec3|] = vec2[i]; } -Dave > On Apr 19, 2017, at 2:24 PM, Dave Crawford wrote: > > Is there a built-in for combining two vectors, or is a for loop the approach? > > As an example: > > vec1 = vector( 1, 2, 3 ); > vec2 = vector( 4, 5, 6 ); > > ...Magic happens here... > > vec3 = (1,2,3,4,5,6) > > > -Dave > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From pssunu6 at gmail.com Thu Apr 20 13:11:28 2017 From: pssunu6 at gmail.com (ps sunu) Date: Fri, 21 Apr 2017 01:41:28 +0530 Subject: [Bro] id into x509.log Message-ID: Hi I need to write id into x509.log , but its giving error redef record X509::Info += { # tx_cc: string &log &optional; #rx_cc: string &log &optional; #tx_asn: count &log &optional; #rx_asn: count &log &optional; id: conn_id &log &optional; }; event file_state_remove(f: fa_file) &priority=5 { if ( ! f$info?$x509 ) return; f$info$x509 = f$id; } type clash in assignment (f$info$x509 = f$id) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170421/cca6a9a1/attachment.html From jazoff at illinois.edu Thu Apr 20 13:25:08 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 20 Apr 2017 20:25:08 +0000 Subject: [Bro] id into x509.log In-Reply-To: References: Message-ID: > On Apr 20, 2017, at 4:11 PM, ps sunu wrote: > > Hi > > I need to write id into x509.log , but its giving error > > redef record X509::Info += { > # tx_cc: string &log &optional; > #rx_cc: string &log &optional; > > #tx_asn: count &log &optional; > #rx_asn: count &log &optional; > id: conn_id &log &optional; > }; > event file_state_remove(f: fa_file) &priority=5 > { > if ( ! f$info?$x509 ) > return; > > f$info$x509 = f$id; > } > x509 info record already has an id field: ## File id of this certificate. id: string &log; and you're trying to assign the entire record to f$id instead of a particular field. Maybe you mean something like this: redef record X509::Info += { conn_id: conn_id &log &optional; }; event file_state_remove(f: fa_file) &priority=5 { if ( ! f$info?$x509 ) return; # Assume this file only has one connection for ( id in f$conns ) local c = f$conns[id]; f$info$x509$conn_id = c$id; } -- - Justin Azoff From jazoff at illinois.edu Thu Apr 20 13:48:03 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 20 Apr 2017 20:48:03 +0000 Subject: [Bro] id into x509.log In-Reply-To: References: Message-ID: <2EF811B3-AE8D-497B-9CDD-B6B6C4FAF25E@illinois.edu> > On Apr 20, 2017, at 4:40 PM, ps sunu wrote: > > yes you are right, i tested above code and its working but not seeing any conn_id related entries > > i am using http://try.bro.org/#/trybro/saved/140090 pcap for testing this Ah right, I tested it with print but didn't actually look at the log. It's a problem with the priority, it's using the same event that is used to log the record, so you need to ensure that your file_state_remove event runs first. I also had the parenthesis slightly wrong (I always screw things up when I try to use the indentation brace style that bro uses) This definitely works: event file_state_remove(f: fa_file) &priority=10 { if ( ! f$info?$x509 ) return; # Assume this file only has one connection for ( id in f$conns ) { local c = f$conns[id]; f$info$x509$conn_id = c$id; } } http://try.bro.org/#/trybro/saved/140102 -- - Justin Azoff From anastasakis62 at gmail.com Fri Apr 21 03:32:36 2017 From: anastasakis62 at gmail.com (mike anastasakis) Date: Fri, 21 Apr 2017 12:32:36 +0200 Subject: [Bro] Speed up bro execution Message-ID: Hello, I am handling rather big pcap files in the size of 500gb and bro execution takes a few hours to complete. For this reason I am looking for ways to speed up the execution. I want to keep only specific logs files with the goal of making my bro execution faster. For my research I want to keep the following files: * conn.log, ssl.log, x509.log, dns.log, http.log* >From what I understood this command should do the trick: *bro -r -b base/protocols/ssl base/protocols/dns base/protocols/conn base/protocols/http* However, with the addition of base/protocol/ssl I also get the tunnel.log and files.log which I do not need. Is there a way to exclude these files from logging? Moreover, I have a rather powerful machines with 8 cores and 8gb of RAM does anyone know a way to fully utilize that when using bro? Thanks all, Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170421/995541b2/attachment.html From hacecky at jlab.org Fri Apr 21 07:49:11 2017 From: hacecky at jlab.org (Eric Hacecky) Date: Fri, 21 Apr 2017 10:49:11 -0400 (EDT) Subject: [Bro] Question about duplicate traffic with load balancing and SSH::Password_Guessing Message-ID: <1378087348.541747.1492786151896.JavaMail.zimbra@jlab.org> I'm trying to correlate the notices generated by SSH::Password_Guessing with bro/firewall logs. I currently have detect-bruteforcing variables at the default of 30 failed SSH attempts over a 30 minute period as the limit before a host is considered to be guessing passwords and a notice is generated. Example: //Bro Notice Message: 1.1.1.1 appears to be guessing SSH passwords (seen in 62 connections). Sub: Sampled servers: 2.2.2.2, 2.2.2.2, 2.2.2.2, 2.2.2.2, 2.2.2.2 (yes it lists the same SSH server 5 times) Src: 1.1.1.1 Dst: - UID: - FUID: - File Mime Type: - File Desc: - Proto: - P: - N: - Peer Descr: worker-2-2 Actions: Notice::ACTION_EMAIL,Notice::ACTION_LOG // Bro ssh.log for that timeframe [root at bro]# cat ssh.21\:00\:00-22\:00\:00.log | /usr/local/bro/bin/bro-cut -d ts id.orig_h auth_success | grep 1.1.1.1 2017-04-18T21:36:58-0400 1.1.1.1 T <--- this line is repeated 31 times 2017-04-18T21:37:45-0400 1.1.1.1 T <--- this line is repeated 31 times Notice that auth_success is True. //Firewall logs Just shows the two (successful) ssh connections at the corresponding times. My load balancing setup: lb_method=myricom lb_procs=31 This is a single box with 32 cores. ---------------- This brings up two questions. Why is SSH:Password_Guessing generating a notice when auth_success is True? Is this expected behavior with my load balancing setup? That the same connection is fed to all 31 cores? Thanks, Eric From jazoff at illinois.edu Fri Apr 21 08:17:40 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 21 Apr 2017 15:17:40 +0000 Subject: [Bro] Question about duplicate traffic with load balancing and SSH::Password_Guessing In-Reply-To: <1378087348.541747.1492786151896.JavaMail.zimbra@jlab.org> References: <1378087348.541747.1492786151896.JavaMail.zimbra@jlab.org> Message-ID: <1ABF943C-BCC6-4EB2-B021-34B3926DF596@illinois.edu> > On Apr 21, 2017, at 10:49 AM, Eric Hacecky wrote: > > My load balancing setup: > > lb_method=myricom > lb_procs=31 > > This is a single box with 32 cores. > > ---------------- > > This brings up two questions. > > Why is SSH:Password_Guessing generating a notice when auth_success is True? > > Is this expected behavior with my load balancing setup? That the same connection is fed to all 31 cores? In order for that configuration to work you need to have the myricom SNF drivers and pcap library installed, and bro must be using that pcap library. If you are seeing the same connection logged 31 times then you are DEFINITELY not using the myricom provided pcap library. If the myricom pcap library is not referenced in /etc/ld.so.conf or similar, you'll need something like this under the worker node: [worker] host=whatever interface=p1p1 lb_method=myricom lb_procs=31 env_vars=LD_PRELOAD=/opt/snf/lib/libsnf.so.0:/opt/snf/lib/libpcap.so.1,SNF_APP_ID=1 -- - Justin Azoff From mabossert at gmail.com Fri Apr 21 09:15:44 2017 From: mabossert at gmail.com (M. Aaron Bossert) Date: Fri, 21 Apr 2017 12:15:44 -0400 Subject: [Bro] Custom log file Message-ID: <586FACED-9746-44C7-B9E6-DC9D347F04D9@gmail.com> I am using bro 2.5 to process PCAP dumps and am storing both the raw PCAP and the bro logs in Hbase. I already have an acceptable pipeline for getting both bro logs and PCAP into Hbase, but I want to be able to have each packet linked back to the conn.log entry (using the uid field). Currently, I am doing this in Hbase, but would rather have bro do it for me. Is it possible to have bro create either individual PCAP files for each log entry or a single log file that listed individual packets (presumably with a packet offset in the PCAP file) along with the uid from the conn.log file? I saw this option in YAF and was hoping it existed in bro. Sent from my iPhone From mabossert at gmail.com Fri Apr 21 09:19:51 2017 From: mabossert at gmail.com (M. Aaron Bossert) Date: Fri, 21 Apr 2017 12:19:51 -0400 Subject: [Bro] [bro] Custom log file References: <586FACED-9746-44C7-B9E6-DC9D347F04D9@gmail.com> Message-ID: <4B731A42-D191-4EEC-AA0A-14E0314202F9@gmail.com> Sorry, forgot to add the [bro] in the subject line... Sent from my iPhone Begin forwarded message: > From: "M. Aaron Bossert" > Date: April 21, 2017 at 12:15:44 EDT > To: bro at bro.org > Subject: Custom log file > > I am using bro 2.5 to process PCAP dumps and am storing both the raw PCAP and the bro logs in Hbase. I already have an acceptable pipeline for getting both bro logs and PCAP into Hbase, but I want to be able to have each packet linked back to the conn.log entry (using the uid field). > > Currently, I am doing this in Hbase, but would rather have bro do it for me. Is it possible to have bro create either individual PCAP files for each log entry or a single log file that listed individual packets (presumably with a packet offset in the PCAP file) along with the uid from the conn.log file? > > I saw this option in YAF and was hoping it existed in bro. > > Sent from my iPhone -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170421/51815512/attachment.html From zwlu at ucdavis.edu Fri Apr 21 16:32:49 2017 From: zwlu at ucdavis.edu (Zhi-Wei Lu) Date: Fri, 21 Apr 2017 23:32:49 +0000 Subject: [Bro] Checking symmetric traffic using bro Message-ID: Hi Bro experts, We are newbie of bro and are in the process of testing a bro setup using Arista 7150 to split traffic using symmetric hashing, sending them to bro cluster. Could bro tell how well the symmetric hashing mechanism is working? What log files/stats shall we look at to discern this information? Justin at bro IRC channel suggested this script https://gist.github.com/JustinAzoff/446d0abba2c6dd8ff242#file-conn-peer-bro This adds peer (worker-x-y) information at the end of conn.log lines, in our case, we have a single bro server with 10 bro workers running on it, this would tell how well bro divide the traffic it received and send to individual workers, is that right? What I am interested in is whether the Arista 7150 split traffic properly so that bro downstream could tell how well or bad traffic was split on Arista? Is that possible? Thank you very much and have a nice weekend. Zhi-Wei Lu IET-CR-Network Operations Center University of California, Davis (530) 752-0155 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170421/93337ed9/attachment.html From bro at pingtrip.com Fri Apr 21 16:35:18 2017 From: bro at pingtrip.com (Dave Crawford) Date: Fri, 21 Apr 2017 19:35:18 -0400 Subject: [Bro] =?utf-8?q?Connection_History=3A_=22connection_direction_was?= =?utf-8?q?_flipped_by_Bro=E2=80=99s_heuristic=22?= Message-ID: What does the caret ("connection direction was flipped by Bro?s heuristic?) in a connections history mean? If the packet in question was spoofed (like the receiving end of a DNS amplification attack) would that trigger Bro?s heuristics? Below are entries from dns, conn and weird logs for the same event for which I can?t find any indications that it sourced from my network. Additionally, there are no subsequent connection attempts to the IP contained the response packet. Dns.log 1491285594.163321 CFXfdl4zMQrM2T15Wa 57555 194.9.69.193 53 udp 21705 - wfuvsrsrwb.www.91duofenxiang[.]com - - - - 0 NOERROR F F F T 0 193.58.251[.]1 60.000000 F Conn.log 1491285594.163321 CFXfdl4zMQrM2T15Wa 57555 194.9.69.193 53 udp dns - - - SHR T ^d 0 0 1 94 (empty) PDC_NSM-4 US RU Weird.log 1491285604.163437 CFXfdl4zMQrM2T15Wa 57555 194.9.69.193 53 dns_unmatched_msg - F PDC_NSM-4 Thanks, -Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170421/74fdbc24/attachment.html From jazoff at illinois.edu Fri Apr 21 19:11:39 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Sat, 22 Apr 2017 02:11:39 +0000 Subject: [Bro] Checking symmetric traffic using bro In-Reply-To: References: Message-ID: > On Apr 21, 2017, at 7:32 PM, Zhi-Wei Lu wrote: > > Hi Bro experts, > > We are newbie of bro and are in the process of testing a bro setup using Arista 7150 to split traffic using symmetric hashing, sending them to bro cluster. Could bro tell how well the symmetric hashing mechanism is working? What log files/stats shall we look at to discern this information? > > Justin at bro IRC channel suggested this script > https://gist.github.com/JustinAzoff/446d0abba2c6dd8ff242#file-conn-peer-bro > > This adds peer (worker-x-y) information at the end of conn.log lines, in our case, we have a single bro server with 10 bro workers running on it, this would tell how well bro divide the traffic it received and send to individual workers, is that right? > > What I am interested in is whether the Arista 7150 split traffic properly so that bro downstream could tell how well or bad traffic was split on Arista? Is that possible? > > Thank you very much and have a nice weekend. I'm a bit confused.. if you only have a single server the switch isn't splitting the traffic at all. Symmetric hashing is only relevant if you have more than one server. For what it's worth, Arista switches in tapagg mode handle symmetric hashing perfectly. -- - Justin Azoff From jazoff at illinois.edu Fri Apr 21 19:23:02 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Sat, 22 Apr 2017 02:23:02 +0000 Subject: [Bro] =?utf-8?q?Connection_History=3A_=22connection_direction_was?= =?utf-8?q?_flipped_by_Bro=E2=80=99s_heuristic=22?= In-Reply-To: References: Message-ID: <60FC9F2A-CB8A-42B3-AB85-9BEDB6C2DD80@illinois.edu> > On Apr 21, 2017, at 7:35 PM, Dave Crawford wrote: > > What does the caret ("connection direction was flipped by Bro?s heuristic?) in a connections history mean? If the packet in question was spoofed (like the receiving end of a DNS amplification attack) would that trigger Bro?s heuristics? Yes.. > > Below are entries from dns, conn and weird logs for the same event for which I can?t find any indications that it sourced from my network. Additionally, there are no subsequent connection attempts to the IP contained the response packet. > > Dns.log > 1491285594.163321 CFXfdl4zMQrM2T15Wa 57555 194.9.69.193 53 udp 21705 - wfuvsrsrwb.www.91duofenxiang[.]com - - - - 0 NOERROR F F F T 0 193.58.251[.]1 60.000000 F > > Conn.log > 1491285594.163321 CFXfdl4zMQrM2T15Wa 57555 194.9.69.193 53 udp dns - - - SHR T ^d 0 0 1 94 (empty) PDC_NSM-4 US RU > > Weird.log > 1491285604.163437 CFXfdl4zMQrM2T15Wa 57555 194.9.69.193 53 dns_unmatched_msg - F PDC_NSM-4 It definitely wasn't sourced from your network. You can see the numbers after the history field(^d) are: orig_pkts=0 orig_ip_bytes=0 resp_pkts=1 resp_ip_bytes=94 which shows that bro saw that you sent 0 packets and received 1. The issue is that you were sent a DNS response packet, which in a perfect world where carriers do proper ingress filtering would have only happened if you had sent the corresponding DNS query packet. Bro assumes it didn't see the query due to capture loss and sets up the orig/resp under that assumption. Unfortunately this doesn't work so well for backscatter, especially when dealing with UDP protocols. It's easy enough to filter out connections like this to another log file if you wanted, generally anything with a local address as a source and resp_pkts=1 and orig_pkts=0 is from backscatter. I've looked into fixing this inside of Bro, but the code that handles this sort of thing is a bit complicated. -- - Justin Azoff From zwlu at ucdavis.edu Fri Apr 21 20:16:21 2017 From: zwlu at ucdavis.edu (Zhi-Wei Lu) Date: Sat, 22 Apr 2017 03:16:21 +0000 Subject: [Bro] Checking symmetric traffic using bro In-Reply-To: References: , Message-ID: Thank you Justin, In our test, the Arista split traffic into two streams, our one bro server analyze only one stream data. Zhi-Wei Lu IET-CR-Network Operations Center University of California, Davis (530) 752-0155 ________________________________ From: Azoff, Justin S Sent: Friday, April 21, 2017 7:11:39 PM To: Zhi-Wei Lu Cc: bro at bro.org Subject: Re: [Bro] Checking symmetric traffic using bro > On Apr 21, 2017, at 7:32 PM, Zhi-Wei Lu wrote: > > Hi Bro experts, > > We are newbie of bro and are in the process of testing a bro setup using Arista 7150 to split traffic using symmetric hashing, sending them to bro cluster. Could bro tell how well the symmetric hashing mechanism is working? What log files/stats shall we look at to discern this information? > > Justin at bro IRC channel suggested this script > https://gist.github.com/JustinAzoff/446d0abba2c6dd8ff242#file-conn-peer-bro > > This adds peer (worker-x-y) information at the end of conn.log lines, in our case, we have a single bro server with 10 bro workers running on it, this would tell how well bro divide the traffic it received and send to individual workers, is that right? > > What I am interested in is whether the Arista 7150 split traffic properly so that bro downstream could tell how well or bad traffic was split on Arista? Is that possible? > > Thank you very much and have a nice weekend. I'm a bit confused.. if you only have a single server the switch isn't splitting the traffic at all. Symmetric hashing is only relevant if you have more than one server. For what it's worth, Arista switches in tapagg mode handle symmetric hashing perfectly. -- - Justin Azoff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170422/751113fb/attachment-0001.html From fatema.bannatwala at gmail.com Mon Apr 24 06:45:29 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Mon, 24 Apr 2017 09:45:29 -0400 Subject: [Bro] Manager swapping.. In-Reply-To: References: <9D6F6FFF-A711-43BC-8FEA-E436F507D6E4@illinois.edu> <8A24D1AD-53B3-46D5-B6D1-A8D3F80FC8B8@illinois.edu> Message-ID: The issue got resolved. :) I rebuilt Bro with tcmalloc, for efficiet memory usage, on the cluster and it seems to resolve the heavy memory usage on the manager. After that, when I disabled the scan scripts in the cluster, the memory usage dropped down to ~5%, and when it's enabled the memory usage toggles around ~25% (i.e ~25-28GB on manager) and around ~31GB on workers, so far the cluster seems to be stable with regard to memory usage. Thank you all for the help, for resolving the issue. Appreciate it :) -Fatema. On Thu, Mar 23, 2017 at 1:24 PM, fatema bannatwala < fatema.bannatwala at gmail.com> wrote: > Thanks Sanjay for suggestions.I already have the @load > protocols/ssl/validate-certs disabled in local.bro. :) > > I was looking into the reporter logs and see some logs like this: > > Some INFO logs: > > 1490288453.884071 Reporter::INFO Got counters: > [new_conn_counter=4394103, is_catch_release_active=7433937, > known_scanners_counter=0, not_scanner=2439888, darknet_counter=64358, > not_darknet_counter=3114626, already_scanner_counter=0, > filteration_entry=0, filteration_success=1543038, > c_knock_filterate=3548445, c_knock_checkscan=0, c_knock_core=0, > c_land_filterate=22317, c_land_checkscan=0, c_land_core=0, > c_backscat_filterate=3548445, c_backscat_checkscan=0, c_backscat_core=0, > c_addressscan_filterate=3548445, c_addressscan_checkscan=0, > c_addressscan_core=0, check_scan_counter=0, worker_to_manager_counter=0, > run_scan_detection=0, check_scan_cache=1543038, event_peer=worker-1-15] > manager > > 1490288454.925040 Reporter::INFO known_scanners_inactive: > [scanner=94.51.38.120, status=T, detection=KnockKnockScan, > detect_ts=1490202054.11266, event_peer=manager, expire=F] manager > 1490288454.925040 Reporter::INFO known_scanners_inactive: > [scanner=171.249.5.188, status=T, detection=KnockKnockScan, > detect_ts=1490202053.07045, event_peer=manager, expire=F] manager > > Ans these error logs: > 0.000000 Reporter::ERROR field value missing > [Scan::geoip_info$country_code] /usr/local/bro/2.5/share/bro/ > site/scan-NG-master/scripts/./scan-summary.bro, line 292 > 0.000000 Reporter::ERROR value used but not set > (Scan::c_landmine_scan_summary) /usr/local/bro/2.5/share/bro/ > site/scan-NG-master/scripts/./check-landmine.bro, line 33 > 0.000000 Reporter::ERROR value used but not set > (Scan::c_landmine_scan_summary) /usr/local/bro/2.5/share/bro/ > site/scan-NG-master/scripts/./check-landmine.bro, line 33 > > Are they anywhere related to the issue? > > Thanks, > Fatema. > > On Thu, Mar 23, 2017 at 10:56 AM, fatema bannatwala < > fatema.bannatwala at gmail.com> wrote: > >> Nope, based on our previous discussion in another thread, >> I disabled the misc/scan, and loaded scan-NG-master script. >> I always thought that the scripts would have more load on workers than >> manager. >> When I was seeing memory issues on workers, I stopped using misc/scan and >> switched to >> the scan-NG script. >> Didn't know that it would impact manager performance as well, hmm. >> >> On Thu, Mar 23, 2017 at 10:43 AM, Azoff, Justin S >> wrote: >> >>> >>> > On Mar 23, 2017, at 7:40 AM, fatema bannatwala < >>> fatema.bannatwala at gmail.com> wrote: >>> > >>> > Thanks Justin for the input :) >>> > >>> > I restarted Bro after disabling some of the protocols logging (like >>> rdp, syslog, snmp etc) yesterday afternoon, >>> > as the machine is in production and needed to be fixed kind of "ASAP". >>> Hence couldn't get a chance to run >>> > the broctl top while having the issue, I know you have mentioned it >>> couple of times in past to use "broctl top" >>> > instead of normal "top", but magically I keep forgetting to do that, I >>> think I should come up with by BRO troubleshoot >>> > guide, which should list some basic troubleshooting commands that you >>> guys suggest in these emails :) >>> > >>> > Anyways, I did run the command today, and it looks like the manager >>> process is overwhelmed, >>> > hmm I thought that it might logger that might be having issues >>> catching up on the load, but I was wrong: >>> > >>> > $ sudo -u bro /usr/local/bro/2.5/bin/broctl top manager logger >>> > Name Type Host Pid Proc VSize Rss Cpu Cmd >>> > logger logger IDS 60928 parent 2G 90M 17% bro >>> > logger logger IDS 60932 child 522M 246M 5% bro >>> > manager manager IDS 60990 child 1G 257M 35% bro >>> > manager manager IDS 60973 parent 222G 31G 23% bro >>> > >>> > It makes me think, if there is some memory leak issue with manager. >>> >>> Are you loading misc/detect-traceroute or misc/scan in your local.bro? >>> >>> -- >>> - Justin Azoff >>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170424/30a6feea/attachment.html From carlopmart at gmail.com Mon Apr 24 07:54:53 2017 From: carlopmart at gmail.com (C. L. Martinez) Date: Mon, 24 Apr 2017 14:54:53 +0000 Subject: [Bro] Some broct cron errors Message-ID: <20170424145453.mxj3nfqywnlbo5jm@scotland.uxdom.org> Hi all, In my Bro manager, under stats.log file I see the following errors: 1493044804.42 worker-2 parent vsize 104857600 1493044804.42 worker-2 parent cmd bro 1493044804.42 worker-2 parent pid 12753 1493044804.42 worker-2 parent cpu 0 1493044804.42 worker-2 parent rss 102760448 1493044804.42 worker-2 child vsize 91226112 1493044804.42 worker-2 child cmd bro 1493044804.42 worker-2 child pid 53186 1493044804.42 worker-2 child cpu 0 1493044804.42 worker-2 child rss 3702784 1493044804.42 worker-2 error error worker-2: capstats failed (error: 'vio0 -i vio2 -i vio3 -i vio4 -i vio7': No such device exists (BIOCSETIF failed: Device not configured)) 1493044804.42 worker-1 error error worker-1: capstats failed (error: 'vio0 -i vio2 -i vio3 -i vio4 -i vio7': No such device exists (BIOCSETIF failed: Device not configured)) I have configured two worker nodes listening in several network interfaces. Executing broctl netstats doesn't return any error: root at brmgr01:/nsm/bro/logs/stats# broctl netstats worker-1: 1493045533.403342 recvd=754293 dropped=0 link=754293 worker-2: 1493045532.906071 recvd=685795 dropped=0 link=685795 I think the problem is with my worker's sections under node.cfg: [worker-1] type=worker host=172.22.59.2 interface='vio0 -i vio2 -i vio3 -i vio4 -i vio7' [worker-2] type=worker host=172.22.59.3 interface='vio0 -i vio2 -i vio3 -i vio4 -i vio7' What am I doing wrong? Bro is 2.5 release ... Thanks -- Greetings, C. L. Martinez From pssunu6 at gmail.com Mon Apr 24 09:05:22 2017 From: pssunu6 at gmail.com (ps sunu) Date: Mon, 24 Apr 2017 21:35:22 +0530 Subject: [Bro] best examples Message-ID: give me suggession of best bro tutorial, examples pdf and scripts Regards, Binu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170424/1e5a05bc/attachment.html From dnthayer at illinois.edu Mon Apr 24 09:53:00 2017 From: dnthayer at illinois.edu (Daniel Thayer) Date: Mon, 24 Apr 2017 11:53:00 -0500 Subject: [Bro] Some broct cron errors In-Reply-To: <20170424145453.mxj3nfqywnlbo5jm@scotland.uxdom.org> References: <20170424145453.mxj3nfqywnlbo5jm@scotland.uxdom.org> Message-ID: <1a0cebba-179b-7007-c2b7-bd5e431d333f@illinois.edu> Capstats does not take multiple interface arguments. If you try to give it multiple interface arguments, it will just use the last one. For example, these two commands are equivalent: capstats -i vio0 -i vio2 capstats -i vio2 The error message you're seeing is because the quotes in your node.cfg are being passed directly to the capstats command (so capstats thinks the interface name is 'vio0 -i vio2 ...'). On 4/24/17 9:54 AM, C. L. Martinez wrote: > Hi all, > > In my Bro manager, under stats.log file I see the following errors: > > 1493044804.42 worker-2 parent vsize 104857600 > 1493044804.42 worker-2 parent cmd bro > 1493044804.42 worker-2 parent pid 12753 > 1493044804.42 worker-2 parent cpu 0 > 1493044804.42 worker-2 parent rss 102760448 > 1493044804.42 worker-2 child vsize 91226112 > 1493044804.42 worker-2 child cmd bro > 1493044804.42 worker-2 child pid 53186 > 1493044804.42 worker-2 child cpu 0 > 1493044804.42 worker-2 child rss 3702784 > 1493044804.42 worker-2 error error worker-2: capstats failed (error: 'vio0 -i vio2 -i vio3 -i vio4 -i vio7': No such device exists (BIOCSETIF failed: Device not configured)) > 1493044804.42 worker-1 error error worker-1: capstats failed (error: 'vio0 -i vio2 -i vio3 -i vio4 -i vio7': No such device exists (BIOCSETIF failed: Device not configured)) > > > I have configured two worker nodes listening in several network interfaces. Executing broctl netstats doesn't return any error: > > root at brmgr01:/nsm/bro/logs/stats# broctl netstats > worker-1: 1493045533.403342 recvd=754293 dropped=0 link=754293 > worker-2: 1493045532.906071 recvd=685795 dropped=0 link=685795 > > I think the problem is with my worker's sections under node.cfg: > > [worker-1] > type=worker > host=172.22.59.2 > interface='vio0 -i vio2 -i vio3 -i vio4 -i vio7' > > [worker-2] > type=worker > host=172.22.59.3 > interface='vio0 -i vio2 -i vio3 -i vio4 -i vio7' > > What am I doing wrong? > > Bro is 2.5 release ... > > Thanks > From zwlu at ucdavis.edu Mon Apr 24 09:57:15 2017 From: zwlu at ucdavis.edu (Zhi-Wei Lu) Date: Mon, 24 Apr 2017 16:57:15 +0000 Subject: [Bro] Checking symmetric traffic using bro In-Reply-To: References: , Message-ID: Hi Justin, We will essentially use a setup like the following https://www.arista.com/assets/data/pdf/Whitepapers/palo_alto_networks_and_arista.pdf We are wondering what tools (bro?) we can use to show that traffic send to each PA-5060 (or whatever IPS tools) from Arista switches are symmetric. Thank you. Zhi-Wei Lu IET-CR-Network Operations Center University of California, Davis (530) 752-0155 From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Zhi-Wei Lu Sent: Friday, April 21, 2017 8:16 PM To: Azoff, Justin S Cc: bro at bro.org Subject: Re: [Bro] Checking symmetric traffic using bro Thank you Justin, In our test, the Arista split traffic into two streams, our one bro server analyze only one stream data. Zhi-Wei Lu IET-CR-Network Operations Center University of California, Davis (530) 752-0155 ________________________________ From: Azoff, Justin S > Sent: Friday, April 21, 2017 7:11:39 PM To: Zhi-Wei Lu Cc: bro at bro.org Subject: Re: [Bro] Checking symmetric traffic using bro > On Apr 21, 2017, at 7:32 PM, Zhi-Wei Lu > wrote: > > Hi Bro experts, > > We are newbie of bro and are in the process of testing a bro setup using Arista 7150 to split traffic using symmetric hashing, sending them to bro cluster. Could bro tell how well the symmetric hashing mechanism is working? What log files/stats shall we look at to discern this information? > > Justin at bro IRC channel suggested this script > https://gist.github.com/JustinAzoff/446d0abba2c6dd8ff242#file-conn-peer-bro > > This adds peer (worker-x-y) information at the end of conn.log lines, in our case, we have a single bro server with 10 bro workers running on it, this would tell how well bro divide the traffic it received and send to individual workers, is that right? > > What I am interested in is whether the Arista 7150 split traffic properly so that bro downstream could tell how well or bad traffic was split on Arista? Is that possible? > > Thank you very much and have a nice weekend. I'm a bit confused.. if you only have a single server the switch isn't splitting the traffic at all. Symmetric hashing is only relevant if you have more than one server. For what it's worth, Arista switches in tapagg mode handle symmetric hashing perfectly. -- - Justin Azoff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170424/77a526d6/attachment-0001.html From zach.rogers at oregonstate.edu Mon Apr 24 15:15:00 2017 From: zach.rogers at oregonstate.edu (Zach Rogers) Date: Mon, 24 Apr 2017 15:15:00 -0700 Subject: [Bro] Packet Loss with Myricom Bro Cluster Message-ID: <1493072100.2796.8.camel@oregonstate.edu> Hello, I have a cluster of five machines for our current Bro configuration: One is acting as the Master & Proxy node, and the other four are Worker nodes. The four worker nodes have Myricom 10g NICs and are using the Bro::Myricom plugin. The workers are getting traffic from the Myricom NICs, though there seems to be substantial packet loss. Here is the output of running 'broctl netstats all', as you can see the dropped packet count is quite high: manager: 1493071687.452948 recvd=5089239 dropped=15719766 link=20825230 proxy-1: 1493071730.166046 recvd=0 dropped=0 link=0 worker-1-1: 1493071533.397463 recvd=16723822 dropped=1827590608 link=1844314430 worker-1-10: 1493071533.598767 recvd=15599277 dropped=1827590608 link=1843189885 worker-1-11: 1493071533.799606 recvd=15290549 dropped=1827590608 link=1842881157 worker-1-12: 1493071534.001465 recvd=17829130 dropped=1827590608 link=1845419738 worker-1-2: 1493071534.201668 recvd=16913145 dropped=1827590608 link=1844503753 worker-1-3: 1493071534.402914 recvd=13317747 dropped=1827590608 link=1840908355 worker-1-4: 1493071534.603851 recvd=16654854 dropped=1827590608 link=1844245462 worker-1-5: 1493071534.791954 recvd=15707229 dropped=1827590608 link=1843297837 worker-1-6: 1493071535.005706 recvd=44924122 dropped=1827590608 link=1872514730 worker-1-7: 1493071535.206737 recvd=15892468 dropped=1827590608 link=1843483076 worker-1-8: 1493071535.407748 recvd=15109664 dropped=1827590608 link=1842700272 worker-1-9: 1493071535.609068 recvd=17725999 dropped=1827590608 link=1845316607 worker-2-1: 1493071452.728863 recvd=12467133 dropped=1654102950 link=1666570083 worker-2-10: 1493071452.930009 recvd=18175352 dropped=1654102950 link=1672278302 worker-2-11: 1493071453.129614 recvd=14050575 dropped=1654102950 link=1668153525 worker-2-12: 1493071453.330973 recvd=13461021 dropped=1654102950 link=1667563971 worker-2-2: 1493071453.532331 recvd=18143473 dropped=1654102950 link=1672246423 worker-2-3: 1493071453.731385 recvd=21263867 dropped=1654102950 link=1675366817 worker-2-4: 1493071453.933757 recvd=21217186 dropped=1654102950 link=1675320136 worker-2-5: 1493071454.135406 recvd=15458489 dropped=1654102950 link=1669561439 worker-2-6: 1493071454.336124 recvd=27904731 dropped=1654102950 link=1682007681 worker-2-7: 1493071454.535881 recvd=15498799 dropped=1654102950 link=1669601749 worker-2-8: 1493071454.738093 recvd=15570231 dropped=1654102950 link=1669673181 worker-2-9: 1493071454.941583 recvd=15546579 dropped=1654102950 link=1669649529 worker-3-1: 1493071595.728039 recvd=16016236 dropped=1393679752 link=1409695988 worker-3-10: 1493071595.928627 recvd=16806524 dropped=1393679752 link=1410486276 worker-3-11: 1493071596.129858 recvd=16960142 dropped=1393679752 link=1410639894 worker-3-12: 1493071596.331357 recvd=19031806 dropped=1393679752 link=1412711558 worker-3-2: 1493071596.531622 recvd=16059903 dropped=1393679752 link=1409739655 worker-3-3: 1493071596.732314 recvd=20528301 dropped=1393679752 link=1414208053 worker-3-4: 1493071596.933870 recvd=17478728 dropped=1393679752 link=1411158480 worker-3-5: 1493071597.134664 recvd=15785290 dropped=1393679752 link=1409465042 worker-3-6: 1493071597.335570 recvd=13051546 dropped=1393679752 link=1406731298 worker-3-7: 1493071597.529927 recvd=17668246 dropped=1393679752 link=1411347998 worker-3-8: 1493071597.736828 recvd=21655950 dropped=1393679752 link=1415335702 worker-3-9: 1493071597.938700 recvd=18237026 dropped=1393679752 link=1411916778 worker-4-1: 1493071949.358339 recvd=19489052 dropped=1504105858 link=1523594910 worker-4-10: 1493071949.559733 recvd=18684571 dropped=1504105858 link=1522790429 worker-4-11: 1493071949.760921 recvd=18578400 dropped=1504105858 link=1522684258 worker-4-12: 1493071949.961847 recvd=15738360 dropped=1504105858 link=1519844218 worker-4-2: 1493071950.160646 recvd=17092444 dropped=1504105858 link=1521198302 worker-4-3: 1493071950.363595 recvd=16890327 dropped=1504105858 link=1520996185 worker-4-4: 1493071950.564875 recvd=14940510 dropped=1504105858 link=1519046368 worker-4-5: 1493071950.765612 recvd=17175752 dropped=1504105858 link=1521281610 worker-4-6: 1493071950.966709 recvd=14854076 dropped=1504105858 link=1518959934 worker-4-7: 1493071951.167789 recvd=15455452 dropped=1504105858 link=1519561310 worker-4-8: 1493071951.368743 recvd=14915366 dropped=1504105858 link=1519021224 worker-4-9: 1493071951.572223 recvd=21105493 dropped=1504105858 link=1525211351 I am looking for guidance on how to figure out the cause of this packet loss, with hopes of fixing the issue. Any ideas? All the best, -- Zach Rogers Security Analyst, Office of Information Security Information Services | Oregon State University http://is.oregonstate.edu/ois From jazoff at illinois.edu Mon Apr 24 15:36:56 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 24 Apr 2017 22:36:56 +0000 Subject: [Bro] Packet Loss with Myricom Bro Cluster In-Reply-To: <1493072100.2796.8.camel@oregonstate.edu> References: <1493072100.2796.8.camel@oregonstate.edu> Message-ID: <46048E7B-DAAE-43C9-963A-F0457A52CDA6@illinois.edu> > On Apr 24, 2017, at 6:15 PM, Zach Rogers wrote: > > Hello, > > I have a cluster of five machines for our current Bro configuration: One > is acting as the Master & Proxy node, and the other four are Worker > nodes. The four worker nodes have Myricom 10g NICs and are using the > Bro::Myricom plugin. > > The workers are getting traffic from the Myricom NICs, though there > seems to be substantial packet loss. Here is the output of running > 'broctl netstats all', as you can see the dropped packet count is quite > high: > worker-1-12: 1493071534.001465 recvd=17829130 dropped=1827590608 > link=1845419738 > ... > I am looking for guidance on how to figure out the cause of this packet > loss, with hopes of fixing the issue. > > Any ideas? > There's an issue with the myricom interface where the dropped packet counter is per card and it does not reset to 0 when the application restarts. This causes the dropped packet counter to appear to be (in your case) at least 12x larger than it is.. If not more, depending on the uptime of the server. Run this to reset the dropped counter and restart bro: broctl stop broctl exec /opt/snf/bin/myri_counters -c -p 0 broctl exec /opt/snf/bin/myri_counters -c -p 1 broctl start That will reset all counters to zero before bro starts and ensure the drop count isn't crazy high before bro even starts. I wrote the attached script a while ago to compute the proper per card drop percentage. It should still work. Some of these issues can possibly be fixed inside of bro/broctl. I think bro can just grab the drop count at startup and if it's non-zero use it as an offset for any future values. The drop count being per card is a bit harder. You could just divide the drop count by the number of workers, but that's really just trading one misleading value for another. -- - Justin Azoff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170424/d094332a/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: netstats_sum.py Type: text/x-python-script Size: 785 bytes Desc: netstats_sum.py Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170424/d094332a/attachment.bin From bill.de.ping at gmail.com Tue Apr 25 04:52:50 2017 From: bill.de.ping at gmail.com (william de ping) Date: Tue, 25 Apr 2017 14:52:50 +0300 Subject: [Bro] write_expire computational time Message-ID: Hi all, After reviewing the source code, I am still not 100% sure how write_expire is implemented. Basically, I want to know if I have a table with n elements, and each element should expire 1 minute after its insertion - will bro loop over all elements in the list checking if they are expired ? if this is the case then write_expire should be O(n), is this correct ? Thank you, B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170425/b99e138c/attachment.html From liburdi.joshua at gmail.com Tue Apr 25 07:06:35 2017 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Tue, 25 Apr 2017 10:06:35 -0400 Subject: [Bro] Hashing incomplete files Message-ID: Hey everyone, Hopefully anyone who has looked at or worked on the hashing component of the file analysis framework can help out with my request. I have a need for Bro to hash all files, including incomplete ones. I looked at the file hashing source code and making Bro hash incomplete files seemed straight forward (comment out the lines that break file hashing if there is an undelivered chunk), but I'm getting an odd result: the hashes reported by Bro for incomplete files are not the same hashes as what is extracted by Bro. For example, here's a files.log entry for an incomplete file with hashing enabled: 1493035575.544634 Fb19KI1OvvCjlT49eg 1.2.3.4 1.2.3.4 CKcFdN2BuVOe1wiFB HTTP 0 EXTRACT,SHA1,MD5 - - 0.036770 - F 32221 59247 27026 0 T - 62f2c17b427ab54f9a8e30f384ba2a5e 6cba20d301dde6d7cbc4f41c689c1ecd108d7bef - extract-1493035575.544634-HTTP-Fb19KI1OvvCjlT49eg Here is the MD5 hash as reported by the file system: f0d987adb1015a05aabfcbade38751b1 extract-1493035575.544634-HTTP-Fb19KI1OvvCjlT49eg Any thoughts on why these hashes don't match? I'm guessing that enabling this functionality isn't as simple as not breaking the hashing function when an undelivered chunk is found. Thanks, Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170425/a9337df6/attachment.html From kmcmahon at mitre.org Tue Apr 25 07:34:41 2017 From: kmcmahon at mitre.org (McMahon, Kevin J) Date: Tue, 25 Apr 2017 14:34:41 +0000 Subject: [Bro] Hashing incomplete files In-Reply-To: References: Message-ID: I?m guessing that Bro doesn?t pass a string of nulls to the hash function when there?s an undelivered chunk. But that?s what ends up in the file (I don?t know if that?s a side effect or intentional ? but it is useful as all the other bits end up in the right place and you can find the holes after the fact). So I wouldn?t expect that the hash would be the same. If you want them to match you probably need to figure out how to pass a block of nulls (of the appropriate length) to the hash function whenever there is undelivered data. Kevin From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Josh Liburdi Sent: Tuesday, April 25, 2017 10:07 AM To: bro Subject: [Bro] Hashing incomplete files Hey everyone, Hopefully anyone who has looked at or worked on the hashing component of the file analysis framework can help out with my request. I have a need for Bro to hash all files, including incomplete ones. I looked at the file hashing source code and making Bro hash incomplete files seemed straight forward (comment out the lines that break file hashing if there is an undelivered chunk), but I'm getting an odd result: the hashes reported by Bro for incomplete files are not the same hashes as what is extracted by Bro. For example, here's a files.log entry for an incomplete file with hashing enabled: 1493035575.544634 Fb19KI1OvvCjlT49eg 1.2.3.4 1.2.3.4 CKcFdN2BuVOe1wiFB HTTP 0 EXTRACT,SHA1,MD5 - - 0.036770 - F 32221 59247 27026 0 T - 62f2c17b427ab54f9a8e30f384ba2a5e 6cba20d301dde6d7cbc4f41c689c1ecd108d7bef - extract-1493035575.544634-HTTP-Fb19KI1OvvCjlT49eg Here is the MD5 hash as reported by the file system: f0d987adb1015a05aabfcbade38751b1 extract-1493035575.544634-HTTP-Fb19KI1OvvCjlT49eg Any thoughts on why these hashes don't match? I'm guessing that enabling this functionality isn't as simple as not breaking the hashing function when an undelivered chunk is found. Thanks, Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170425/a2788cdc/attachment-0001.html From johanna at icir.org Tue Apr 25 07:49:33 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 25 Apr 2017 07:49:33 -0700 Subject: [Bro] Hashing incomplete files In-Reply-To: References: Message-ID: <20170425144933.sbf425sqrddlqepk@Beezling.local> On Tue, Apr 25, 2017 at 02:34:41PM +0000, McMahon, Kevin J wrote: > I?m guessing that Bro doesn?t pass a string of nulls to the hash > function when there?s an undelivered chunk. But that?s what ends up in > the file (I don?t know if that?s a side effect or intentional ? but it > is useful as all the other bits end up in the right place and you can > find the holes after the fact). So I wouldn?t expect that the hash > would be the same. Just to add a bit to this - I think this behavior is intentional and used, e.g., when a file is downloaded from over multiple streams simultaneously. Johanna From johanna at icir.org Tue Apr 25 07:54:39 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 25 Apr 2017 07:54:39 -0700 Subject: [Bro] write_expire computational time In-Reply-To: References: Message-ID: <20170425145439.buxrdfcctqum46w2@Beezling.local> Hi, the main function implementing expiry is TableVal::DoExpire in Val.cc (approximately line 2175). > Basically, I want to know if I have a table with n elements, and each > element should expire 1 minute after its insertion - will bro loop over all > elements in the list checking if they are expired ? Yes, Bro will loop over all elements from time to time, setting internal timeouts that cause a loop over the whole table removing expired elements. Note that elements are not guaranteed to expire after the expiration time; they will be removed sometime after expiration time, but it can take a bit. > if this is the case then write_expire should be O(n), is this correct ? The time overhead of expiration is O(n), correct. Johanna From liburdi.joshua at gmail.com Tue Apr 25 07:55:29 2017 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Tue, 25 Apr 2017 10:55:29 -0400 Subject: [Bro] Hashing incomplete files In-Reply-To: <20170425144933.sbf425sqrddlqepk@Beezling.local> References: <20170425144933.sbf425sqrddlqepk@Beezling.local> Message-ID: Kevin was correct -- filling in the incomplete space with nulls produces the same MD5 hash. Johanna, in the case of an "incomplete" file, could multiple simultaneous streams produce an inconsistent hash? Not sure I understand how multiple streams might affect a file's completeness, but would happy to hear your thoughts. Josh On Tue, Apr 25, 2017 at 10:49 AM, Johanna Amann wrote: > On Tue, Apr 25, 2017 at 02:34:41PM +0000, McMahon, Kevin J wrote: > > I?m guessing that Bro doesn?t pass a string of nulls to the hash > > function when there?s an undelivered chunk. But that?s what ends up in > > the file (I don?t know if that?s a side effect or intentional ? but it > > is useful as all the other bits end up in the right place and you can > > find the holes after the fact). So I wouldn?t expect that the hash > > would be the same. > > Just to add a bit to this - I think this behavior is intentional and used, > e.g., when a file is downloaded from over multiple streams simultaneously. > > Johanna > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170425/1790e244/attachment.html From johanna at icir.org Tue Apr 25 07:57:16 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 25 Apr 2017 07:57:16 -0700 Subject: [Bro] Checking symmetric traffic using bro In-Reply-To: References: Message-ID: <20170425145716.zsqqddyk7hqwfciy@Beezling.local> Without looking at the document, usually the easy way to tell if traffic is symmetric is to check if connection history in conn.log has a lot of lines that are all capitalized or all non-capitalized (which shows that a sensor only received one side of the connections). This can be combined with the script that you mentioned earlier if you are running in cluster mode. Johanna On Mon, Apr 24, 2017 at 04:57:15PM +0000, Zhi-Wei Lu wrote: > Hi Justin, > > We will essentially use a setup like the following > https://www.arista.com/assets/data/pdf/Whitepapers/palo_alto_networks_and_arista.pdf > > We are wondering what tools (bro?) we can use to show that traffic send to each PA-5060 (or whatever IPS tools) from Arista switches are symmetric. > > Thank you. > > Zhi-Wei Lu > IET-CR-Network Operations Center > University of California, Davis > (530) 752-0155 > > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Zhi-Wei Lu > Sent: Friday, April 21, 2017 8:16 PM > To: Azoff, Justin S > Cc: bro at bro.org > Subject: Re: [Bro] Checking symmetric traffic using bro > > > Thank you Justin, > > > > In our test, the Arista split traffic into two streams, our one bro server analyze only one stream data. > > > Zhi-Wei Lu > > IET-CR-Network Operations Center > University of California, Davis > (530) 752-0155 > ________________________________ > From: Azoff, Justin S > > Sent: Friday, April 21, 2017 7:11:39 PM > To: Zhi-Wei Lu > Cc: bro at bro.org > Subject: Re: [Bro] Checking symmetric traffic using bro > > > > On Apr 21, 2017, at 7:32 PM, Zhi-Wei Lu > wrote: > > > > Hi Bro experts, > > > > We are newbie of bro and are in the process of testing a bro setup using Arista 7150 to split traffic using symmetric hashing, sending them to bro cluster. Could bro tell how well the symmetric hashing mechanism is working? What log files/stats shall we look at to discern this information? > > > > Justin at bro IRC channel suggested this script > > https://gist.github.com/JustinAzoff/446d0abba2c6dd8ff242#file-conn-peer-bro > > > > This adds peer (worker-x-y) information at the end of conn.log lines, in our case, we have a single bro server with 10 bro workers running on it, this would tell how well bro divide the traffic it received and send to individual workers, is that right? > > > > What I am interested in is whether the Arista 7150 split traffic properly so that bro downstream could tell how well or bad traffic was split on Arista? Is that possible? > > > > Thank you very much and have a nice weekend. > > I'm a bit confused.. if you only have a single server the switch isn't splitting the traffic at all. Symmetric hashing is only relevant if you have more than one server. > > For what it's worth, Arista switches in tapagg mode handle symmetric hashing perfectly. > > > -- > - Justin Azoff > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Tue Apr 25 07:58:47 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 25 Apr 2017 07:58:47 -0700 Subject: [Bro] best examples In-Reply-To: References: Message-ID: <20170425145847.aj6kk7kvafaeil6o@Beezling.local> You might already be aware, but the most thorough list that we have simply is at https://www.bro.org/documentation. Johanna On Mon, Apr 24, 2017 at 09:35:22PM +0530, ps sunu wrote: > give me suggession of best bro tutorial, examples pdf and scripts > > > Regards, > Binu > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Tue Apr 25 08:01:04 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 25 Apr 2017 08:01:04 -0700 Subject: [Bro] Manager swapping.. In-Reply-To: References: <9D6F6FFF-A711-43BC-8FEA-E436F507D6E4@illinois.edu> <8A24D1AD-53B3-46D5-B6D1-A8D3F80FC8B8@illinois.edu> Message-ID: <20170425150104.5fnsd7x5xnygqxhc@Beezling.local> Oh, that is interesting. Just to check - this was on 2.5? Would you potentially be up for a little bit of digging to see what is causing this? I am not aware of anyone encountering this problem before, and I really would like this not to happen :) If you are ok with it, I will supply you with an instrumented version of the validation script that outputs a bit of debugging information to help me check what is going on here. Johanna On Thu, Mar 23, 2017 at 03:40:32PM +0000, Hovsep Levi wrote: > Try disabling the SSL/TLS cert verification. I'm not sure why but > that helped, without it the manager would slowly climb to massive > memory usage. Now it works fine for one or two weeks before > unexpectedly using all memory. > > #@load protocols/ssl/validate-certs > > Good: > > Name Type Host Pid Proc VSize Rss Cpu Cmd > logger-1 logger 10.1.1.1 6241 parent 701M 163M 20% bro > logger-1 logger 10.1.1.1 6261 child 458M 69M 3% bro > manager manager 10.1.1.1 6345 child 510M 377M 100% bro > manager manager 10.1.1.1 6292 parent 890M 804M 24% bro > > > Bad: > > Name Type Host Pid Proc VSize Rss Cpu Cmd > logger-1 logger 10.1.1.1 52731 parent 1G 806M 0% bro > logger-1 logger 10.1.1.1 52951 child 8G 8G 0% bro > manager manager 10.1.1.1 53127 child 1G 742M 0% bro > manager manager 10.1.1.1 52979 parent 1573G 100G 0% bro > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From robin at icir.org Tue Apr 25 08:07:06 2017 From: robin at icir.org (Robin Sommer) Date: Tue, 25 Apr 2017 08:07:06 -0700 Subject: [Bro] write_expire computational time In-Reply-To: <20170425145439.buxrdfcctqum46w2@Beezling.local> References: <20170425145439.buxrdfcctqum46w2@Beezling.local> Message-ID: <20170425150706.GA95899@icir.org> On Tue, Apr 25, 2017 at 07:54 -0700, Johanna Amann wrote: > > if this is the case then write_expire should be O(n), is this correct > > ? > > The time overhead of expiration is O(n), correct. To clarify because there could be different interpretations here: it's indeed O(n) for the whole expire operation, however that is amortized over a longer time frame: when Bro iterates over the table, it works on short slices at a time (which is the reason that it can take longer to expire an element, as Johanna wrote). So it's not O(n) for each table operation or such. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From johanna at icir.org Tue Apr 25 08:07:53 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 25 Apr 2017 08:07:53 -0700 Subject: [Bro] [bro] Custom log file In-Reply-To: <4B731A42-D191-4EEC-AA0A-14E0314202F9@gmail.com> References: <586FACED-9746-44C7-B9E6-DC9D347F04D9@gmail.com> <4B731A42-D191-4EEC-AA0A-14E0314202F9@gmail.com> Message-ID: <20170425150753.czgnojjusgj26kht@Beezling.local> On Fri, Apr 21, 2017 at 12:19:51PM -0400, M. Aaron Bossert wrote: > Sorry, forgot to add the [bro] in the subject line... It actually is added automatically :) > > I am using bro 2.5 to process PCAP dumps and am storing both the raw > > PCAP and the bro logs in Hbase. I already have an acceptable pipeline > > for getting both bro logs and PCAP into Hbase, but I want to be able > > to have each packet linked back to the conn.log entry (using the uid > > field). > > Currently, I am doing this in Hbase, but would rather have bro do it > > for me. Is it possible to have bro create either individual PCAP > > files for each log entry or a single log file that listed individual > > packets (presumably with a packet offset in the PCAP file) along with > > the uid from the conn.log file? Let me prefix this with the caveat of "this is probably not a good idea, because Bro is not built for this use-case". That being said - I don't think that Bro exposes the packet offset of a pcap file, so that is out. However, it should indeed be possible to create individual pcap files for each conn.log entry. The way I would try doing this is catching the new_packet event https://www.bro.org/sphinx/scripts/base/bif/event.bif.bro.html#id-new_packet and then using dump_current_packet to write the packet to a pcap file https://www.bro.org/sphinx/scripts/base/bif/bro.bif.bro.html#id-dump_current_packet (dump_current_packet appends to already existing filess). Note that I did not test if this really works and this probably carries a hefty performance penalty. Johanna From johanna at icir.org Tue Apr 25 08:09:38 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 25 Apr 2017 08:09:38 -0700 Subject: [Bro] Combining Vectors In-Reply-To: <48FBC4B2-8EE7-4253-8B42-535038F35B73@pingtrip.com> References: <9236AEAD-A85B-4D73-817A-28F10F6103DC@pingtrip.com> <48FBC4B2-8EE7-4253-8B42-535038F35B73@pingtrip.com> Message-ID: <20170425150938.4ct6ac4k37o3majj@Beezling.local> Hi Dave, I am not aware of a merge built-in for combining vectors; so your approach probably is the best there is. Johanna On Thu, Apr 20, 2017 at 01:32:34PM -0400, Dave Crawford wrote: > This is how I implemented the ?merge? I needed: > > for (i in vec1) { vec3[|vec3|] = vec1[i]; } > for (i in vec2) { vec3[|vec3|] = vec2[i]; } > > -Dave > > > On Apr 19, 2017, at 2:24 PM, Dave Crawford wrote: > > > > Is there a built-in for combining two vectors, or is a for loop the approach? > > > > As an example: > > > > vec1 = vector( 1, 2, 3 ); > > vec2 = vector( 4, 5, 6 ); > > > > ...Magic happens here... > > > > vec3 = (1,2,3,4,5,6) > > > > > > -Dave > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From asharma at lbl.gov Tue Apr 25 08:18:52 2017 From: asharma at lbl.gov (Aashish Sharma) Date: Tue, 25 Apr 2017 08:18:52 -0700 Subject: [Bro] write_expire computational time In-Reply-To: <20170425145439.buxrdfcctqum46w2@Beezling.local> References: <20170425145439.buxrdfcctqum46w2@Beezling.local> Message-ID: <20170425151851.GV74675@mac-822.local> I have a couple of bro policies in production where I store/expire/extend hundreds of thousands (if not million+) elements/records from table(s). SO far has been operationally workable. Offcourse, the expirations don't happen at the dot on the clock but often little later but that doesn't concern much. Aashish On Tue, Apr 25, 2017 at 07:54:39AM -0700, Johanna Amann wrote: > Hi, > > the main function implementing expiry is TableVal::DoExpire in Val.cc > (approximately line 2175). > > > Basically, I want to know if I have a table with n elements, and each > > element should expire 1 minute after its insertion - will bro loop over all > > elements in the list checking if they are expired ? > > Yes, Bro will loop over all elements from time to time, setting internal > timeouts that cause a loop over the whole table removing expired elements. > Note that elements are not guaranteed to expire after the expiration time; > they will be removed sometime after expiration time, but it can take a > bit. > > > if this is the case then write_expire should be O(n), is this correct ? > > The time overhead of expiration is O(n), correct. > > Johanna > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Tue Apr 25 08:20:25 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 25 Apr 2017 08:20:25 -0700 Subject: [Bro] - set\table\vector types have a complexity of O(n) ? In-Reply-To: <0388a196-23bc-17fb-3382-a8b6a2c12779@gmail.com> References: <0388a196-23bc-17fb-3382-a8b6a2c12779@gmail.com> Message-ID: <20170425152025.dgbi635qxwlvzhtt@Beezling.local> On Tue, Apr 18, 2017 at 01:49:08PM +0200, Jan Grash?fer wrote: > > Just wondering, sets\tables\vectors all have a read\write complexity of > > O(n) ? > > n - referring to the number of elements in the container. > > If I am not mistaken, sets as well as tables are implemented as hash > tables. Thus the average complexity for lookup and insert is O(1). > Vectors are implemented using C++ vectors, I think. I.e., lookup would > be O(1) while inserting depends on the context. Just to (partially) confirm this - tables (and sets, which just are a special case of tables) are implemented as hash-tables with O(1) for insert/lookup. Bro has its own hash-table implementation and is not using the STL for this. Vectors are implemented using stl vectors; the complexity depends on the implementation, but generally lookups should be O(1), and inserts can be O(n) in the worst case, since they might require a copy of the whole data structure, if additional space has to be allocated. Johanna > > Jan > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From asharma at lbl.gov Tue Apr 25 08:24:12 2017 From: asharma at lbl.gov (Aashish Sharma) Date: Tue, 25 Apr 2017 08:24:12 -0700 Subject: [Bro] write_expire computational time In-Reply-To: <20170425150706.GA95899@icir.org> References: <20170425145439.buxrdfcctqum46w2@Beezling.local> <20170425150706.GA95899@icir.org> Message-ID: <20170425152410.GW74675@mac-822.local> Oh! good point. That also reminds me that if your expire times are < 10seconds then you'd have to redef table_expire_interval to a value < 10 seconds. bro ./test-expire-func.bro 1483204866.281666 starting: table_iteration, 1 1483204866.281932 inside table_expire_func: table_iteration, 2 1483204876.381928 inside table_expire_func: table_iteration, 3 1483204886.481921 inside table_expire_func: table_iteration, 4 1483204896.582114 inside table_expire_func: table_iteration, 5 Note: expire kicks in every 10 seconds. ## Check for expired table entries after this amount of time. ## ## .. bro:see:: table_incremental_step table_expire_delay const table_expire_interval = 10 secs &redef; Now reducing to < 10 seconds and putting a million elements is a tradeoff. So measure on your end before making production. Aashish On Tue, Apr 25, 2017 at 08:07:06AM -0700, Robin Sommer wrote: > > > On Tue, Apr 25, 2017 at 07:54 -0700, Johanna Amann wrote: > > > > if this is the case then write_expire should be O(n), is this correct > > > ? > > > > The time overhead of expiration is O(n), correct. > > To clarify because there could be different interpretations here: it's > indeed O(n) for the whole expire operation, however that is amortized > over a longer time frame: when Bro iterates over the table, it works > on short slices at a time (which is the reason that it can take longer > to expire an element, as Johanna wrote). So it's not O(n) for each > table operation or such. > > Robin > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jan.grashoefer at gmail.com Tue Apr 25 08:29:27 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Tue, 25 Apr 2017 17:29:27 +0200 Subject: [Bro] write_expire computational time In-Reply-To: <20170425152410.GW74675@mac-822.local> References: <20170425145439.buxrdfcctqum46w2@Beezling.local> <20170425150706.GA95899@icir.org> <20170425152410.GW74675@mac-822.local> Message-ID: <3769c9b5-b7bf-ad9e-541a-4e33c21d4a0d@gmail.com> > Oh! good point. That also reminds me that if your expire times are < 10seconds then you'd have to redef table_expire_interval to a value < 10 seconds. I was just typing the same! You can also influence the delay and step size using table_expire_interval and table_incremental_step (https://www.bro.org/sphinx/scripts/base/init-bare.bro.html?#id-table_expire_delay). Jan From liburdi.joshua at gmail.com Tue Apr 25 08:56:27 2017 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Tue, 25 Apr 2017 11:56:27 -0400 Subject: [Bro] Hashing incomplete files In-Reply-To: References: <20170425144933.sbf425sqrddlqepk@Beezling.local> Message-ID: Actually, I think I understand what you mean now -- with some of the PCAPs I have, the hash for incomplete files changes from run to run. On Tue, Apr 25, 2017 at 10:55 AM, Josh Liburdi wrote: > Kevin was correct -- filling in the incomplete space with nulls produces > the same MD5 hash. > > Johanna, in the case of an "incomplete" file, could multiple simultaneous > streams produce an inconsistent hash? Not sure I understand how multiple > streams might affect a file's completeness, but would happy to hear your > thoughts. > > Josh > > On Tue, Apr 25, 2017 at 10:49 AM, Johanna Amann wrote: > >> On Tue, Apr 25, 2017 at 02:34:41PM +0000, McMahon, Kevin J wrote: >> > I?m guessing that Bro doesn?t pass a string of nulls to the hash >> > function when there?s an undelivered chunk. But that?s what ends up in >> > the file (I don?t know if that?s a side effect or intentional ? but it >> > is useful as all the other bits end up in the right place and you can >> > find the holes after the fact). So I wouldn?t expect that the hash >> > would be the same. >> >> Just to add a bit to this - I think this behavior is intentional and used, >> e.g., when a file is downloaded from over multiple streams simultaneously. >> >> Johanna >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170425/8d86f6ea/attachment.html From liburdi.joshua at gmail.com Tue Apr 25 10:43:06 2017 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Tue, 25 Apr 2017 13:43:06 -0400 Subject: [Bro] Hashing incomplete files In-Reply-To: References: <20170425144933.sbf425sqrddlqepk@Beezling.local> Message-ID: I take it back, my code was off -- seems fine now. It would be nice if this could be enabled as an option via an argument. On Tue, Apr 25, 2017 at 11:56 AM, Josh Liburdi wrote: > Actually, I think I understand what you mean now -- with some of the PCAPs > I have, the hash for incomplete files changes from run to run. > > On Tue, Apr 25, 2017 at 10:55 AM, Josh Liburdi > wrote: > >> Kevin was correct -- filling in the incomplete space with nulls produces >> the same MD5 hash. >> >> Johanna, in the case of an "incomplete" file, could multiple simultaneous >> streams produce an inconsistent hash? Not sure I understand how multiple >> streams might affect a file's completeness, but would happy to hear your >> thoughts. >> >> Josh >> >> On Tue, Apr 25, 2017 at 10:49 AM, Johanna Amann wrote: >> >>> On Tue, Apr 25, 2017 at 02:34:41PM +0000, McMahon, Kevin J wrote: >>> > I?m guessing that Bro doesn?t pass a string of nulls to the hash >>> > function when there?s an undelivered chunk. But that?s what ends up in >>> > the file (I don?t know if that?s a side effect or intentional ? but it >>> > is useful as all the other bits end up in the right place and you can >>> > find the holes after the fact). So I wouldn?t expect that the hash >>> > would be the same. >>> >>> Just to add a bit to this - I think this behavior is intentional and >>> used, >>> e.g., when a file is downloaded from over multiple streams >>> simultaneously. >>> >>> Johanna >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170425/425845d7/attachment-0001.html From espressobeanies at gmail.com Tue Apr 25 13:45:33 2017 From: espressobeanies at gmail.com (Espresso Beanies) Date: Tue, 25 Apr 2017 16:45:33 -0400 Subject: [Bro] Changing notice log entry actions from Action::Log to Action::Email Message-ID: Hi, In searching previous Bro posts, I'm still not able to understand how to get Bro to email certain notice types as opposed to just creating log entries. My local.bro file contains the following: redef Notice::emailed_types += { > TeamCymruMalwareHashRegistry::Match, > Intel::Notice, > Intel::DOMAIN, > Intel::CERT_HASH, > Intel::FILE_HASH, > }; > redef Notice::type_suppression_intervals += { > [TeamCymruMalwareHashRegistry::Match] = 1hr, > [Intel::Notice] = 1hr, > [Intel::DOMAIN] = 1hr, > [Intel::CERT_HASH] = 1hr, > [Intel::FILE_HASH] = 1hr, > }; Based on this, I'm assuming I would be receiving a summary of all the defined Notice::emailed_types every hour by email. Instead, I'm only receiving Connection Summaries, [Bro] Crash reports, and PacketFilter::Dropped_Packets. If I open my notice.log I see the following: #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path notice #open 2017-04-25-16-00-22 #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p fuid file_mime_type file_desc proto note msg sub src dst p n peer_descr actions suppress_for dropped remote_location.country_code remote_location.region remote_location.city remote_locat ion.latitude remote_location.longitude #types time string addr port addr port string string string enum enum string string addr addr port count string set[enum] interval bool string string string double double 1493150418.640398 - - - - - - - - - SSH::Password_Guessing XXX.XXX.XXX.XXX appears to be guessing SSH pass words (seen in 41 connections). Sampled servers: XXX.XXX.XXX.XXX, XXX.XXX.XXX.XXX, XXX.XXX.XXX.XXX, XXX.XXX.XXX.XXX, XXX.XXX.XXX.XXX XXX.XXX.XXX.XXX - - - worker-2-9 Notice::ACTION_LOG 3600.000000 F - - - - - 1493150706.509497 - - - - - - - - - SSH::Password_Guessing XXX.XXX.XXX.XXX appears to be guessing SSH passw ords (seen in 34 connections). Sampled servers: XXX.XXX.XXX.XXX, XXX.XXX.XXX.XXX, XXX.XXX.XXX.XXX, XXX.XXX.XXX.XXX, XXX.XXX.XXX.XXX XXX.XXX.XXX.XXX - - - worker-2-3 Notice::ACTION_LOG 3600.000000 F - - - - - 1493150707.543255 - - - - - - - - - HTTP::SQL_Injection_Attacker An SQL injection attacker was discover ed! - XXX.XXX.XXX.XXX - - - worker-1-11 Notice::ACTION_LOG 3600.000000 F - - - - - 1493151025.415982 - - - - - - - - - CaptureLoss::Too_Much_Loss The capture loss script detected an es timated loss rate above 24.664% - - - - - worker-2-6 Notice::ACTION_LOG 3600.000000 F - - - - - 1493151925.408827 - - - - - - - - - CaptureLoss::Too_Much_Loss The capture loss script detected an es timated loss rate above 35.923% - - - - - worker-2-5 Notice::ACTION_LOG 3600.000000 F - - - - For these entries, where or what file do I change specific Notice::Types from Notice::ACTION_LOG to Notice::ACTION_EMAIL? Thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170425/28c701c8/attachment.html From jazoff at illinois.edu Tue Apr 25 14:16:12 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 25 Apr 2017 21:16:12 +0000 Subject: [Bro] Changing notice log entry actions from Action::Log to Action::Email In-Reply-To: References: Message-ID: <766320E0-11B2-4C40-A44E-2C5988606253@illinois.edu> > On Apr 25, 2017, at 4:45 PM, Espresso Beanies wrote: > > Hi, > > In searching previous Bro posts, I'm still not able to understand how to get Bro to email certain notice types as opposed to just creating log entries. > > My local.bro file contains the following: > > redef Notice::emailed_types += { > TeamCymruMalwareHashRegistry::Match, > Intel::Notice, > Intel::DOMAIN, > Intel::CERT_HASH, > Intel::FILE_HASH, > }; ... > For these entries, where or what file do I change specific Notice::Types from Notice::ACTION_LOG to Notice::ACTION_EMAIL? The Notice::emailed_types that is in your local.bro that you included in your email. If you want to get emailed about SSH::Password_Guessing then it should be in the emailed_types set. https://www.bro.org/sphinx/frameworks/notice.html#notice-policy-shortcuts -- - Justin Azoff From franky.meier.1 at gmx.de Tue Apr 25 23:14:39 2017 From: franky.meier.1 at gmx.de (Frank Meier) Date: Wed, 26 Apr 2017 08:14:39 +0200 Subject: [Bro] script to extract elastic search mapping from header of bro-logs Message-ID: <20170426081439.127e59cd@NB181106> Hello, many of us use Elastic Search as a sink for bro-logs. I am thinking about written a script to extract the correct mapping from the bro header. This would mean: * mapping data types: string, addr, enum -> string int, count, port -> long interval, double -> double time -> epoch_millis * setting 'not_analyzed' for types like addr where this makes no sense * handle container types (table, set, vector) Any ideas? Has anyone done this before? Franky From johanna at icir.org Wed Apr 26 05:10:04 2017 From: johanna at icir.org (Johanna Amann) Date: Wed, 26 Apr 2017 05:10:04 -0700 Subject: [Bro] script to extract elastic search mapping from header of bro-logs In-Reply-To: <20170426081439.127e59cd@NB181106> References: <20170426081439.127e59cd@NB181106> Message-ID: <20170426121004.cckchz7oxwz5zkto@Beezling.local> Hi, in case you are talking about importing a Bro ASCII log into the database - I did something like that for Postgres once. My script automatically created tables with the right types (including stuff like inet), and converted sets and vectors to postgres arrays. Source is at https://github.com/0xxon/bro-utils Johanna On Wed, Apr 26, 2017 at 08:14:39AM +0200, Frank Meier wrote: > Hello, > > many of us use Elastic Search as a sink for bro-logs. I am thinking > about written a script to extract the correct mapping from the bro > header. > > This would mean: > * mapping data types: > string, addr, enum -> string > int, count, port -> long > interval, double -> double > time -> epoch_millis > * setting 'not_analyzed' for types like addr where this makes no sense > * handle container types (table, set, vector) > > Any ideas? Has anyone done this before? > > Franky > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From espressobeanies at gmail.com Wed Apr 26 08:36:56 2017 From: espressobeanies at gmail.com (Espresso Beanies) Date: Wed, 26 Apr 2017 11:36:56 -0400 Subject: [Bro] Changing notice log entry actions from Action::Log to Action::Email In-Reply-To: <766320E0-11B2-4C40-A44E-2C5988606253@illinois.edu> References: <766320E0-11B2-4C40-A44E-2C5988606253@illinois.edu> Message-ID: Ah, got it. Thanks Justin. On Tue, Apr 25, 2017 at 5:16 PM, Azoff, Justin S wrote: > > > On Apr 25, 2017, at 4:45 PM, Espresso Beanies > wrote: > > > > Hi, > > > > In searching previous Bro posts, I'm still not able to understand how to > get Bro to email certain notice types as opposed to just creating log > entries. > > > > My local.bro file contains the following: > > > > redef Notice::emailed_types += { > > TeamCymruMalwareHashRegistry::Match, > > Intel::Notice, > > Intel::DOMAIN, > > Intel::CERT_HASH, > > Intel::FILE_HASH, > > }; > > ... > > > For these entries, where or what file do I change specific Notice::Types > from Notice::ACTION_LOG to Notice::ACTION_EMAIL? > > The Notice::emailed_types that is in your local.bro that you included in > your email. > > If you want to get emailed about SSH::Password_Guessing then it should be > in the emailed_types set. > > https://www.bro.org/sphinx/frameworks/notice.html#notice-policy-shortcuts > > -- > - Justin Azoff > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170426/cf066885/attachment-0001.html From damian.miller at gmail.com Wed Apr 26 10:34:39 2017 From: damian.miller at gmail.com (Damian Miller) Date: Wed, 26 Apr 2017 10:34:39 -0700 Subject: [Bro] standalone bro performance Message-ID: Hi There, Does anyone have any more recent performance data for standalone Bro ? The most recent I have found is from 2015 where the conclusion is 3600pps (indicated on page 16) Thanks in advance. https://www.sans.org/reading-room/whitepapers/intrusion/ open-source-ids-high-performance-shootout-35772 Damian -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170426/8d8faac8/attachment.html From ed.sealing at sealingtech.org Wed Apr 26 21:31:13 2017 From: ed.sealing at sealingtech.org (Ed Sealing) Date: Thu, 27 Apr 2017 00:31:13 -0400 Subject: [Bro] Speed up bro execution In-Reply-To: References: Message-ID: There are two ways you may be able to address the issues of maximizing processor use, but both would take some additional work. May be worth it if you need to do this a lot. 1. Setup a TCPREPLAY server that forwards the traffic to the bro system and use the normal fanout options (AF_PACKET, PF_RING, etc) to maximize processor usage. You could theoretically do the replay and listening on the same box, but tcpreplay would likely bog down one of the procs (2 more for the manager/proxy, leaving 5 for Bro analysis). 2.Divide up your 500gb PCAP into smaller chunks (e.g. different pcaps based on SRC/DST) and run "bro -r /path/to/pcap" multiple times in parallel. You'd have to point each run to a different log-file path and combine them later. Best to write a script for this if you often have to run large pcaps offline. On Fri, Apr 21, 2017 at 6:32 AM, mike anastasakis wrote: > Hello, > > I am handling rather big pcap files in the size of 500gb and bro execution > takes a few hours to complete. For this reason I am looking for ways to > speed up the execution. > > I want to keep only specific logs files with the goal of making my bro > execution faster. For my research I want to keep the following files: * > conn.log, ssl.log, x509.log, dns.log, http.log* > From what I understood this command should do the trick: *bro -r > -b base/protocols/ssl base/protocols/dns > base/protocols/conn base/protocols/http* > However, with the addition of base/protocol/ssl I also get the tunnel.log > and files.log which I do not need. Is there a way to exclude these files > from logging? > > > Moreover, I have a rather powerful machines with 8 cores and 8gb of RAM > does anyone know a way to fully utilize that when using bro? > > > Thanks all, > Mike > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- R/S *Ed Sealing President / CEO* *CISSP, CEH, RHCSA* 7134 Columbia Gateway Dr Suite 160 Columbia, MD 21046 Mobile: (301) 885-6947 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170427/50d69984/attachment.html From franky.meier.1 at gmx.de Thu Apr 27 04:55:08 2017 From: franky.meier.1 at gmx.de (Frank Meier) Date: Thu, 27 Apr 2017 13:55:08 +0200 Subject: [Bro] script to extract elastic search mapping from header of bro-logs In-Reply-To: <20170426121004.cckchz7oxwz5zkto@Beezling.local> References: <20170426081439.127e59cd@NB181106> <20170426121004.cckchz7oxwz5zkto@Beezling.local> Message-ID: <20170427135508.0bbee503@NB181106> Hi, On Wed, 26 Apr 2017 05:10:04 -0700 Johanna Amann wrote: > Hi, > > in case you are talking about importing a Bro ASCII log into the > database > - I did something like that for Postgres once. My script automatically > created tables with the right types (including stuff like inet), and > converted sets and vectors to postgres arrays. > thanks, that's what I was thinking about. Franky From seth at corelight.com Thu Apr 27 11:17:18 2017 From: seth at corelight.com (Seth Hall) Date: Thu, 27 Apr 2017 14:17:18 -0400 Subject: [Bro] standalone bro performance In-Reply-To: References: Message-ID: <21A5DA3E-7E92-41C6-8552-C53DAEE401E5@corelight.com> > On Apr 26, 2017, at 1:34 PM, Damian Miller wrote: > > Does anyone have any more recent performance data for standalone Bro ? The most recent I have found is from 2015 where the conclusion is 3600pps (indicated on page 16) > https://www.sans.org/reading-room/whitepapers/intrusion/open-source-ids-high-performance-shootout-35772 I don't believe that it's possible to do comparisons between software like this, but that paper is pretty funny because they are referring to a research paper from 2007. I suppose you probably remember what computers were like in 2007... quite a bit slower than today. Please use the assumption that those numbers are wildly wrong now. :) .Seth -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From vladg at illinois.edu Fri Apr 28 07:55:05 2017 From: vladg at illinois.edu (Vlad Grigorescu) Date: Fri, 28 Apr 2017 09:55:05 -0500 Subject: [Bro] script to extract elastic search mapping from header of bro-logs In-Reply-To: <201704271259.v3RCxJaq013924@vladg.net> References: <20170426081439.127e59cd@NB181106> <20170426121004.cckchz7oxwz5zkto@Beezling.local> <201704271259.v3RCxJaq013924@vladg.net> Message-ID: ElasticSearch gets difficult, because there's a lot of context-specific data that should be captured too, especially when it comes to indexing. For example, I liked to index domain names with a reverse-path tokenization on '.' as the delimeter, so that www.ncsa.illinois.edu will show up in searches for "edu," "illinois.edu," "ncsa.illinois.edu," and "www.ncsa.illinois.edu." Capturing this context can be very tricky, and I don't think that it's currently available in the ASCII logs. I'd be curious if anyone has thoughts on how to improve this. --Vlad Frank Meier writes: > Hi, > > On Wed, 26 Apr 2017 05:10:04 -0700 Johanna Amann > wrote: > >> Hi, >> >> in case you are talking about importing a Bro ASCII log into the >> database >> - I did something like that for Postgres once. My script automatically >> created tables with the right types (including stuff like inet), and >> converted sets and vectors to postgres arrays. >> > > thanks, that's what I was thinking about. > > Franky > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 800 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170428/454f776b/attachment.bin From vladg at illinois.edu Fri Apr 28 08:24:42 2017 From: vladg at illinois.edu (Vlad Grigorescu) Date: Fri, 28 Apr 2017 10:24:42 -0500 Subject: [Bro] PySubnetTree on PyPI In-Reply-To: <201704181841.v3IIfKnW010297@vladg.net> References: <201704181841.v3IIfKnW010297@vladg.net> Message-ID: Hi George, Thanks for the reminder. I've uploaded the newer versions to PyPI. Please let me know if you run into any issues. Thanks, --Vlad George Macon writes: > The PySubnetTree library was uploaded to PyPI in 2014 when it was on > version 0.23; this is still the most recent version on PyPI. This had > been originally requested in the GitHub Issue #1, which I note was never > closed. I asked on IRC where the appropriate place to ask about getting > the most recent version uploaded and was directed to the mailing list. > Can whoever controls the "bro" account on PyPI upload the newest version > of PySubnetTree? > > Thanks, > George > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 800 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170428/e1754bc9/attachment.bin From jlay at slave-tothe-box.net Fri Apr 28 08:51:59 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 28 Apr 2017 09:51:59 -0600 Subject: [Bro] Get bro to listen only to localhost Message-ID: Interestingly, a Nessus scan now crashes bro...I can replicate this on one machine, so, how do I tell bro not to listen to all interfaces, and just localhost? It's a standalone instance. Thank you. James From pssunu6 at gmail.com Fri Apr 28 08:59:33 2017 From: pssunu6 at gmail.com (ps sunu) Date: Fri, 28 Apr 2017 21:29:33 +0530 Subject: [Bro] Tor script Message-ID: Hi any new tor detection script available in bro language ? Regards, Sunu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170428/d1014426/attachment.html From jazoff at illinois.edu Fri Apr 28 09:01:02 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 28 Apr 2017 16:01:02 +0000 Subject: [Bro] Get bro to listen only to localhost In-Reply-To: References: Message-ID: <433BB6CB-A944-4113-A510-878DC869276F@illinois.edu> > On Apr 28, 2017, at 11:51 AM, James Lay wrote: > > Interestingly, a Nessus scan now crashes bro...I can replicate this on > one machine, so, how do I tell bro not to listen to all interfaces, and > just localhost? It's a standalone instance. Thank you. > > James Can you report that as a bug? I'm not sure if it is an option and broctl should set up bro to listen on localhost by default in standalone mode anyway. -- - Justin Azoff From jlay at slave-tothe-box.net Fri Apr 28 09:47:49 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 28 Apr 2017 10:47:49 -0600 Subject: [Bro] Get bro to listen only to localhost In-Reply-To: <433BB6CB-A944-4113-A510-878DC869276F@illinois.edu> References: <433BB6CB-A944-4113-A510-878DC869276F@illinois.edu> Message-ID: <32cdb736221fd5bc10bad7b69e631ffb@localhost> Can do..thanks Justin! James On 2017-04-28 10:01, Azoff, Justin S wrote: >> On Apr 28, 2017, at 11:51 AM, James Lay >> wrote: >> >> Interestingly, a Nessus scan now crashes bro...I can replicate this on >> one machine, so, how do I tell bro not to listen to all interfaces, >> and >> just localhost? It's a standalone instance. Thank you. >> >> James > > Can you report that as a bug? I'm not sure if it is an option and > broctl should set up bro to listen on localhost by default in > standalone mode anyway. From pssunu6 at gmail.com Fri Apr 28 11:13:37 2017 From: pssunu6 at gmail.com (ps sunu) Date: Fri, 28 Apr 2017 23:43:37 +0530 Subject: [Bro] bro_init Message-ID: Hi, Any way to insert any connection related details inside event bro_init() functions Regards, sunu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170428/44d1abb1/attachment.html From garg.anant at gmail.com Fri Apr 28 13:52:16 2017 From: garg.anant at gmail.com (anant garg) Date: Fri, 28 Apr 2017 13:52:16 -0700 Subject: [Bro] Cross compiling bro for mips core (as a binary) Message-ID: Hello there, Has anybody got success in cross compiling bro for mips core, specifically as Cavium's Octeon binary (simple executive) ? I have looked around but did not find any pointer on this information. Does not look straightforward to me. Can somebody help providing any information/tips/notes on this if you have tried it before ? I appreciate your time on helping out this. -Anant From pssunu6 at gmail.com Sat Apr 29 03:06:11 2017 From: pssunu6 at gmail.com (ps sunu) Date: Sat, 29 Apr 2017 15:36:11 +0530 Subject: [Bro] (no subject) Message-ID: Hi below script i need to add connection related info inside notice log Skip to content This repository Search Pull requests Issues Gist @binups Sign out Watch 14 Star 29 Fork 10 fox-it/bro-scripts Code Issues 1 Pull requests 0 Projects 0 Wiki Pulse Graphs Branch: master Find file Copy pathbro-scripts/smb-ransomware/smb-ransomware.bro b3aa56d on 12 Sep 2016 @fox-srt fox-srt Adding SMB Ransomware policy for BroCon 2016 1 contributor RawBlameHistory 120 lines (93 sloc) 2.85 KB @load base/frameworks/files @load base/protocols/smb @load base/frameworks/notice @load base/frameworks/sumstats global fuidmap : set[string]; module FoxCryptoRansom; export { redef enum Notice::Type += { ## Notice corresponding to a possible ransomware attack RANSOMWARE_SMB }; ## Entropy check on the first packet send const enc_off = 0 &redef; ## Entropy check on certain bytes const enc_sdata = 0 &redef; const enc_edata = 1000 &redef; ## Entropy and Mean corresponding to a possible ransomware attack const enc_entropy = 7.5 &redef; const enc_mean = 125 &redef; ## Notice values corresponding to a possible ransomware attack const threshold_time = 30sec &redef; const threshold_limit = 5.0 &redef; ## Ignore list for certain filenames const ignore_list = /GoogleChrome/ &redef; redef enum Log::ID += {LOG}; type Info: record { ts: time &log; filename: string &log; entropy: double &log; mean: double &log; }; } event chunk_event (f: fa_file, data: string, off: count) { if ( off == enc_off ) { local fox_entropy = find_entropy(data[enc_sdata:enc_edata]); if ( fox_entropy$entropy >= enc_entropy && fox_entropy$mean >= enc_mean ) { SumStats::observe("SMB traffic detected", SumStats::Key(), SumStats::Observation($num=1)); local rec: FoxCryptoRansom::Info = [ $ts=network_time(), $filename = f$info$filename, $entropy=fox_entropy$entropy, $mean=fox_entropy$mean ]; Log::write(FoxCryptoRansom::LOG, rec); } } } event file_over_new_connection(f: fa_file, c: connection, is_orig: bool) { if (f$source == "SMB"){ local filename = "UNKNOWN"; if (f$info?$filename){ filename = f$info$filename; } local mime_type = "UNKNOWN"; if (f$info?$mime_type){ mime_type = f$info$mime_type; } if (mime_type == "UNKNOWN"){ if (ignore_list in filename){ return; }else{ if ( ! c$smb_state$current_file?$action){ return ; }else{ if (c$smb_state$current_file$action == SMB::FILE_WRITE){ local fuid = c$smb_state$current_file$fuid; if (fuid !in fuidmap){ Files::add_analyzer(f, Files::ANALYZER_DATA_EVENT, Files::AnalyzerArgs($chunk_event=chunk_event)); add fuidmap[fuid]; } }else{ } } } } } } event bro_init() { local r1 = SumStats::Reducer($stream="SMB traffic detected", $apply=set(SumStats::SUM)); SumStats::create([$name = "Ransomware detection", $epoch = threshold_time, $reducers = set(r1), $threshold = threshold_limit, $threshold_val(key: SumStats::Key, result: SumStats::Result) = { return result["SMB traffic detected"]$sum; }, $threshold_crossed(key: SumStats::Key, result: SumStats::Result) = { NOTICE([$note=RANSOMWARE_SMB, $msg="Ransomware encrypting share detected"]); }]); Log::create_stream(FoxCryptoRansom::LOG, [$columns=Info, $path="entropy"]); } -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170429/e3e9b45b/attachment.html From pssunu6 at gmail.com Sun Apr 30 10:36:58 2017 From: pssunu6 at gmail.com (ps sunu) Date: Sun, 30 Apr 2017 23:06:58 +0530 Subject: [Bro] files.log Message-ID: Hi , This method can we add id into files.log global myevent: event(f: fa_file, c: connection, is_orig: bool); redef record Files::Info += { # tx_cc: string &log &optional; #rx_cc: string &log &optional; #tx_asn: count &log &optional; #rx_asn: count &log &optional; id: conn_id &log &optional; }; event myevent(f: fa_file, c: connection, is_orig: bool) &priority = -10 { if ( ! f?$info ) return; f$info$id = c$id; } Regards, Sunub event bro_init() { event myevent( f: fa_file, c: connection, is_orig: bool); } event bro_done() { print "bro_done()"; } -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170430/671d8b6b/attachment.html