From jchambers at ucla.edu Tue Jun 2 12:09:39 2009 From: jchambers at ucla.edu (Jason Chambers) Date: Tue, 02 Jun 2009 12:09:39 -0700 Subject: [Bro] Hardware Experience In-Reply-To: <4A1ED430.2020500@andrew.cmu.edu> References: <8A6C3E52-76BD-46F5-B210-051D2D5392F4@illinois.edu> <4A1DA88F.1060603@ucar.edu> <20090528042100.GE12888@uiuc.edu> <4A1ED13E.4080504@ucla.edu> <4A1ED430.2020500@andrew.cmu.edu> Message-ID: <4A2578F3.1040300@ucla.edu> Something I found out about these cards is they are PCIe v1.1.. "PCIe v2.0 compatible" doesn't mean what I thought it did. So even with a PCIe v2.0 system you can only get 12.5 Gbps from the card. No idea at the moment when they will have a 2.0 version. --Jason Jason Carr wrote: > One thing I noticed with the NT20E is that the web site states that "20 > Gbps throughput @ 64 bytes". I'm assuming that this means that the > device only captures 64 bytes of the data section of a packet. I also > assume this is configurable. For some things that's fine, but in most > NIDS (such as Bro, snort, etc) you usually want the whole packet. > > What are you using in terms of capture size and bandwidth, if you don't > mind me asking? > > - Jason > > Jason Chambers wrote: >> Martin Holste wrote: >>> Your DAG experience is interesting. We demoed the 6.2SE's and they >>> seemed to run OK on libpcap apps for a few days in late 2006. We've >>> been running the smaller 1 Gb cousin, the 4.5G2, in production since >>> then with zero stability problems with libpcap apps. Link size is 1 Gb >>> physical, 450 Mb/sec typical load. In my experience though, the >>> difference maker is rarely in getting the packets to the CPU, but rather >>> in the CPU grepping through the packets fast enough. I anticipate that >>> the Bro cluster work will do more for full snaplength processing than >>> hardware acceleration will unless someone writes Bro for Nvidia's CUDA >>> like they wrote Snort for CUDA with Gnort. >>> >> I recommend these cards available from nPulse networks. [1] (Napatech is >> the OEM). They have more features than the Endace cards and twice the >> port density. And, they fully support FreeBSD. Despite my numerous >> requests it seems Endace maintains that there will not be future support >> for FreeBSD due to lack of demand. To the best of my knowledge, the >> last official supported FreeBSD version from Endace is the 6.x train. >> Anyhow that's my personal gripe. >> >> >> [1] http://www.npulsenetworks.com/ >> >> Napatech 2x10GE NT20E >> >> http://www.napatech.com/products/capture_adapters/2x10g_pcie_nt20e.html >> >> >> And when it's available, the NTNPU20E looks like a very exciting >> complement to the NT20E's. It was displayed at Interop but is still a >> few months out from release. >> >> http://www.napatech.com/products/inspect_adapters.html >> >> >> HTH, >> >> --Jason >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > -- Jason Chambers UCLA jchambers at ucla.edu 310-206-5603 From lothar at lobraun.de Mon Jun 8 07:01:37 2009 From: lothar at lobraun.de (Lothar Braun) Date: Mon, 08 Jun 2009 16:01:37 +0200 Subject: [Bro] Artificial SYN-Packets? Message-ID: <4A2D19C1.2080409@lobraun.de> Hi all, I wrote a bro script that works on the flags in the TCP header and on the identifier field in IP header. While some TCP connections can be processed without any problems, others seem to produce strange results with my script. The attached pcap file (example.pcap) contains a problematic connection. As you can see this starts with four SYN-Packets (probably due to retransmits) which also have ECN and CWR set. The identifier field of this packets is set a custom 0x3fff. If you run bro-1.4 with the attached script (test.bro), which prints the id-field and the flags, you will get this output: $ bin/bro -C -r example.pcap test.bro 0 2 16383 210 16383 208 16383 216 16383 210 16383 208 16383 216 8191 216 As you can see only one SYN-Packet has been passed to new_packet() in the script. And this packet does neither transport the correct id nor the correct flags. I think this problem only occurs when the first SYN packet has been retransmitted. My questions: 1.) Is it the desired behavior to only pass one SYN-Packet to new_packet() instead of all SYN-Packets? In my opinion it might be a good idea to get all packets, that have been transmitted (or observed). 2.) Is it desired behavior that the passed SYN-packet does not contain all the information that have been in the original packet? 3.) Can I tune bro to give me the original packet? Best regards, Lothar -------------- next part -------------- A non-text attachment was scrubbed... Name: example.pcap Type: application/octet-stream Size: 2057 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20090608/61af7d00/attachment.obj -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: test.bro Url: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20090608/61af7d00/attachment.ksh From vern at icir.org Mon Jun 8 07:31:45 2009 From: vern at icir.org (Vern Paxson) Date: Mon, 08 Jun 2009 07:31:45 -0700 Subject: [Bro] Artificial SYN-Packets? In-Reply-To: <4A2D19C1.2080409@lobraun.de> (Mon, 08 Jun 2009 16:01:37 +0200). Message-ID: <200906081431.n58EVn2N030232@pork.ICSI.Berkeley.EDU> A bunch of the packets have bad TCP checksums. This is likely the problem - the event engine is discarding them on that account. Vern From lothar at lobraun.de Mon Jun 8 07:49:31 2009 From: lothar at lobraun.de (Lothar Braun) Date: Mon, 08 Jun 2009 16:49:31 +0200 Subject: [Bro] Artificial SYN-Packets? In-Reply-To: <200906081431.n58EVn2N030232@pork.ICSI.Berkeley.EDU> References: <200906081431.n58EVn2N030232@pork.ICSI.Berkeley.EDU> Message-ID: <4A2D24FB.4030209@lobraun.de> Hi Vern, Vern Paxson wrote: > A bunch of the packets have bad TCP checksums. This is likely the problem - > the event engine is discarding them on that account. Thank you for the quick reply. All the packets have bad checksums, because I padded them with tcprewrite and forgot to use the fix checksum option. I therefore used bro -C to disable checksum testing when I ran my script (bro actually would have discarded these packets without -C). So I don't think this is the source of the problem. To make sure, I fixed the checksums with tcprewrite (see attached pcap) and still get the same problem. Best regards, Lothar -------------- next part -------------- A non-text attachment was scrubbed... Name: example-fixed.pcap Type: application/octet-stream Size: 2057 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20090608/f62a9f19/attachment.obj From rmkml at free.fr Mon Jun 8 08:35:03 2009 From: rmkml at free.fr (rmkml) Date: Mon, 8 Jun 2009 17:35:03 +0200 (CEST) Subject: [Bro] Artificial SYN-Packets? In-Reply-To: <4A2D24FB.4030209@lobraun.de> References: <200906081431.n58EVn2N030232@pork.ICSI.Berkeley.EDU> <4A2D24FB.4030209@lobraun.de> Message-ID: Hi Lothar, I you read your last network pcap trace with wireshark, you have same+multiple first Syn (and WE flags) tcp packet, bro understand simply tcp retransmit? confirmed by conn.log: 1022404512.136083 ? 194.44.56.35 139.103.147.106 http 59235 80 tcp 361 ? S1 X and packet number 9 are duplicate, and packets number 10 and 11 are retransmit. Regards Rmkml Crusoe-Researches.com On Mon, 8 Jun 2009, Lothar Braun wrote: > Hi Vern, > > Vern Paxson wrote: >> A bunch of the packets have bad TCP checksums. This is likely the problem - >> the event engine is discarding them on that account. > > Thank you for the quick reply. > > All the packets have bad checksums, because I padded them with > tcprewrite and forgot to use the fix checksum option. I therefore used > bro -C to disable checksum testing when I ran my script (bro actually > would have discarded these packets without -C). So I don't think this is > the source of the problem. > > To make sure, I fixed the checksums with tcprewrite (see attached pcap) > and still get the same problem. > > Best regards, > Lothar > From vern at icir.org Tue Jun 9 04:42:08 2009 From: vern at icir.org (Vern Paxson) Date: Tue, 09 Jun 2009 04:42:08 -0700 Subject: [Bro] Artificial SYN-Packets? In-Reply-To: <4A2D24FB.4030209@lobraun.de> (Mon, 08 Jun 2009 16:49:31 +0200). Message-ID: <200906091142.n59BgCoh014872@pork.ICSI.Berkeley.EDU> Okay, I analyzed this, and the answer is that the connection compressor can't generate new_packet events for some packets because new_packet requires an associated connection (first parameter to the event handler), and the point of the compressor is to not initially create connections. It can't really fake up a new_packet event in this context once it does create the connection, because it has (deliberately) lost the interesting detail. Your script should work as expected if you run it with use_connection_compressor=F. Perhaps the presence of a new_packet event handler should turn off the compressor automatically; or perhaps we should change new_packet to not have an associated connection (though I imagine that would often prove inconvenient). Vern From Tyler.Schoenke at colorado.edu Tue Jun 9 09:57:02 2009 From: Tyler.Schoenke at colorado.edu (Tyler T. Schoenke) Date: Tue, 09 Jun 2009 10:57:02 -0600 Subject: [Bro] Stand-alone cluster problems Message-ID: <4A2E945E.4070704@colorado.edu> Hi, Just wondering if anyone has run into this problem. I'm running Robin's 1.4 cluster code on a stand-alone AMD dual-core machine that is monitoring a 200 Mbit connection. I've been running this setup for a couple months, and it has been working well. I noticed that Bro seemed to be missing some packets, and Seth told me to look for DroppedPackets in the notice.log. From what I recall, I was dropping upwards of 90% of the packets after filtering. The strange thing was the primary bro process CPU usage seemed to be low (20-30%), even though it was dropping most of the packets. I would have expected CPU to be high in trying to keep up. Is there some throttling mechanism to prevent the CPU from being maxed out? I turned on some restrict filters to bring the DroppedPackets down. I turned off various things like HTTP, HTTPS, DNS, IMAP, SMTP, and a few others. After that, the ratio of (packets dropped after filtering) to received was less than one percent. So that is the history and here is my problem. When I start the cluster, the above mentioned ratio will be less than 1%. It remains less than 1% for several days to a week. For no explicable reason, it will jump up as high as 99% and stay stuck there. This has happened twice in the last couple weeks. Whatever caused this problem also caused the logs to stop being rotated. They are showing a timestamp of June 7th (MST). When I run a cluster status, it shows the cluster is running, and I can see both bro-1.4-robin processes running, but their CPU usage has dropped down to 0.00%. I think the CPU usage for one of those processes had typically been around 16-20% when the dropped ratio was less than 1%. Restarting the cluster should clear the problem again for a few days. Is there any other troubleshooting I can do before restarting to determine the cause of the problem? Below are a few lines showing the high ratio of dropped packets. These are some of the last lines logged, so based on the timestamps, everything stopped around 01:48 6/8/09 GMT. 1244425212.229043:DroppedPackets:NOTICE_FILE:bro::::::::::317741 packets dropped after filtering, 387325 received, 1117174 on link::@26-c775-10c8ed 1244425222.229050:DroppedPackets:NOTICE_FILE:bro::::::::::16000 packets dropped after filtering, 68358 received, 192124 on link::@26-c775-10c91b 1244425236.737126:DroppedPackets:NOTICE_FILE:bro::::::::::311615 packets dropped after filtering, 341963 received, 979510 on link::@26-c775-10c939 1244425247.488693:DroppedPackets:NOTICE_FILE:bro::::::::::79770 packets dropped after filtering, 172843 received, 477468 on link::@26-c775-10c981 1244425261.942659:DroppedPackets:NOTICE_FILE:bro::::::::::315110 packets dropped after filtering, 315358 received, 878859 on link::@26-c775-10c987 1244425690.711860:DroppedPackets:NOTICE_FILE:bro::::::::::10878167 packets dropped after filtering, 10878305 received, 31316595 on link::@26-c775-10c994 Here are some earlier logs showing what the ratio normally looks like. 1243649373.579172:DroppedPackets:NOTICE_FILE:bro::::::::::1752 packets dropped after filtering, 333996 received, 612544 on link::@88-1bd3-bd5af 1243649383.579295:DroppedPackets:NOTICE_FILE:bro::::::::::1333 packets dropped after filtering, 342521 received, 627890 on link::@88-1bd3-bd5c2 1243649393.579339:DroppedPackets:NOTICE_FILE:bro::::::::::920 packets dropped after filtering, 326511 received, 605614 on link::@88-1bd3-bd5d1 1243649403.579424:DroppedPackets:NOTICE_FILE:bro::::::::::1336 packets dropped after filtering, 318679 received, 615016 on link::@88-1bd3-bd5ec Tyler -- Tyler Schoenke IT Security Office University of Colorado - Boulder From robin at icir.org Wed Jun 10 10:22:32 2009 From: robin at icir.org (Robin Sommer) Date: Wed, 10 Jun 2009 10:22:32 -0700 Subject: [Bro] Stand-alone cluster problems In-Reply-To: <4A2E945E.4070704@colorado.edu> References: <4A2E945E.4070704@colorado.edu> Message-ID: <20090610172232.GA86099@icir.org> Hi Tyler, if I understand you correctly, there are actually two problems: - Bro is dropping many packets even when running at rather low CPU - after a few days, Bro hangs with 99% CPU and stalls. Is that correct? Regarding the former, generally at 20-30% CPU Bro shouldn't drop any signficant amount of packets, there's no throttling mechanism or such. One guess here would be the operating system. What kind of system are you running on? Have you tried the tuning described on http://www.net.t-labs.tu-berlin.de/research/bpcs/? Another question: is there any regularity in the timestamps of when the drops occur? Like in regular intervals? (But longer intervals than 10s as that's just the reporting interval). Regarding the latter, it would be good to know in which part of the code Bro hangs. To find that out, can do you the following next time it happens: - Attach a gdb with "gdb /path/to/bro/binary process-id-of-the-main-Bro-process" - Send me a stack backstrace from gdb's "bt" command I wouldn't be totally surprised if the state checkpointing is the culprit. To test that, can you remove the line "@load checkpoint" from cluster.bro? Robin -- Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From adayadil.thomas at gmail.com Thu Jun 11 07:38:47 2009 From: adayadil.thomas at gmail.com (Adayadil Thomas) Date: Thu, 11 Jun 2009 10:38:47 -0400 Subject: [Bro] SSH login brute force Message-ID: Greetings. Does bro detect SSH brute force login attempts? I saw that there was ssh and ssh-stepping policies, and they do not seem to be looking for any kind of brute force detection, but I wanted to ask the group about this. Thanks for the reply, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20090611/19ab7eb8/attachment.html From hall.692 at osu.edu Thu Jun 11 08:29:44 2009 From: hall.692 at osu.edu (Seth Hall) Date: Thu, 11 Jun 2009 11:29:44 -0400 Subject: [Bro] SSH login brute force In-Reply-To: References: Message-ID: On Jun 11, 2009, at 10:38 AM, Adayadil Thomas wrote: > Does bro detect SSH brute force login attempts? My ssh-ext.bro script at the following link does, but it could certainly be improved. http://github.com/sethhall/bro_scripts/tree/master .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From adayadil.thomas at gmail.com Thu Jun 11 08:48:58 2009 From: adayadil.thomas at gmail.com (Adayadil Thomas) Date: Thu, 11 Jun 2009 11:48:58 -0400 Subject: [Bro] SSH login brute force In-Reply-To: References: Message-ID: Thanks for the info, Seth Can you point me to any info/document/link that you may have used for your approach. for e.g. about how you set authentication_data_size = 5500 I am trying to understand how a brute force attempt can be distinguished from a normal client server communication since both are encrypted? On Thu, Jun 11, 2009 at 11:29 AM, Seth Hall wrote: > > On Jun 11, 2009, at 10:38 AM, Adayadil Thomas wrote: > >> Does bro detect SSH brute force login attempts? > > My ssh-ext.bro script at the following link does, but it could certainly be improved. > > http://github.com/sethhall/bro_scripts/tree/master > > ?.Seth > > --- > Seth Hall > Network Security - Office of the CIO > The Ohio State University > Phone: 614-292-9721 > From hall.692 at osu.edu Thu Jun 11 10:44:01 2009 From: hall.692 at osu.edu (Seth Hall) Date: Thu, 11 Jun 2009 13:44:01 -0400 Subject: [Bro] SSH login brute force In-Reply-To: References: Message-ID: It heuristically makes a guess about a successful login based on the amount of data returned from the server (the default is 5k). It's far from 100% accurate, but in my environment has been extremely useful. Recently, I've been looking through some SSH traces trying to find a more refined heuristic because if someone logs in and then logs out again right away, it's likely the server will cross the byte count threshold and a successful connection will be marked as unsuccessful. If you have any ideas for how to make that happen, I'd be glad to hear. .Seth On Jun 11, 2009, at 11:48 AM, Adayadil Thomas wrote: > Thanks for the info, Seth > > Can you point me to any info/document/link that you may have used for > your approach. > for e.g. about how you set > authentication_data_size = 5500 > > I am trying to understand how a brute force attempt can be > distinguished from a normal > client server communication since both are encrypted? > > > > > > > On Thu, Jun 11, 2009 at 11:29 AM, Seth Hall wrote: >> >> On Jun 11, 2009, at 10:38 AM, Adayadil Thomas wrote: >> >>> Does bro detect SSH brute force login attempts? >> >> My ssh-ext.bro script at the following link does, but it could >> certainly be improved. >> >> http://github.com/sethhall/bro_scripts/tree/master >> >> .Seth >> >> --- >> Seth Hall >> Network Security - Office of the CIO >> The Ohio State University >> Phone: 614-292-9721 >> --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From hall.692 at osu.edu Thu Jun 11 10:50:05 2009 From: hall.692 at osu.edu (Seth Hall) Date: Thu, 11 Jun 2009 13:50:05 -0400 Subject: [Bro] SSH login brute force In-Reply-To: References: Message-ID: <2219B351-FDA0-445F-8FBE-144C537F83F4@osu.edu> On Jun 11, 2009, at 1:44 PM, Seth Hall wrote: > Recently, I've been looking through some SSH traces trying to find a > more refined heuristic because if someone logs in and then logs out > again right away, it's likely the server will cross the byte count > threshold and a successful connection will be marked as unsuccessful. Oops, make that "...will *not* cross the byte count..." .Seth From Tyler.Schoenke at colorado.edu Fri Jun 12 10:35:41 2009 From: Tyler.Schoenke at colorado.edu (Tyler T. Schoenke) Date: Fri, 12 Jun 2009 11:35:41 -0600 Subject: [Bro] Stand-alone cluster problems In-Reply-To: <20090610172232.GA86099@icir.org> References: <4A2E945E.4070704@colorado.edu> <20090610172232.GA86099@icir.org> Message-ID: <4A3291ED.2070807@colorado.edu> Robin Sommer wrote: > if I understand you correctly, there are actually two problems: > > - Bro is dropping many packets even when running at rather low CPU Hi Robin, Yes, that is the way it seemed when I didn't have restrict filters turned on. When the cluster started, the CPU for the Bro process would be high, but would drop down to 20-40% even though many packets were being dropped after filtering. > - after a few days, Bro hangs with 99% CPU and stalls. Partially correct. Bro appears to be hanging, but the CPU is at 0%, and the DroppedPackets/received ratio was banging against 99% just before it started to hang. I haven't restarted the cluster yet, so here is the backtrace. Lines from top: PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 51061 XXXXXX 1 -20 0 1207M 843M swread 1 606:53 0.00% bro-1.4-rob 51082 XXXXXX 1 44 5 31556K 228K select 0 19:29 0.00% bro-1.4-rob I tried attaching to the process with the large TIME value. Is that the primary one? $gdb `which bro-1.4-robin` 51061 (gdb) bt #0 0x081d0e96 in free (mem=0xd724e28) at malloc.c:4229 #1 0x285cfc01 in operator delete () from /usr/lib/libstdc++.so.6 #2 0x080a8f0a in ~Dictionary (this=0x99cd4a0) at Dict.cc:101 #3 0x081c7348 in ~TableEntryValPDict (this=0x99cd4a0) at Val.h:49 #4 0x081c42ac in ~TableVal (this=0x99cd408) at Val.cc:1697 #5 0x081c0e28 in TableVal::DoExpire (this=0x8669d60, t=1244434191.756459) at Obj.h:213 #6 0x081a9be2 in PQ_TimerMgr::DoAdvance (this=0x82f2a18, new_t=1244434191.756459, max_expire=300) at Timer.cc:164 #7 0x0813ff09 in expire_timers (src_ps=0x90495a0) at Net.cc:392 #8 0x0813ffbd in net_packet_dispatch (t=1244434191.756459, hdr=0x90495d8, pkt=0x9049a6a "", hdr_size=14, src_ps=0x90495a0, pkt_elem=0x0) at Net.cc:412 #9 0x08140549 in net_packet_arrival (t=1244434191.756459, hdr=0x90495d8, pkt=0x9049a6a "", hdr_size=14, src_ps=0x90495a0) at Net.cc:496 #10 0x0814ef1f in PktSrc::Process (this=0x90495a0) at PktSrc.cc:199 #11 0x081402b5 in net_run () at Net.cc:526 #12 0x080501be in main (argc=454545480, argv=0xbfbfeb28) at main.cc:1056 Here is the bt from the other process just in case it helps. $gdb `which bro-1.4-robin` 51082 (gdb) bt #0 0x286f8da3 in select () from /lib/libc.so.7 #1 0x081617fa in SocketComm::Run (this=0xbfbfe770) at RemoteSerializer.cc:2743 #2 0x0816629a in RemoteSerializer::Fork (this=0x82fa580) at RemoteSerializer.cc:600 #3 0x081664aa in RemoteSerializer::Init (this=0x82fa580) at RemoteSerializer.cc:525 #4 0x0804fbab in main (argc=-2147483647, argv=0xbfbfeb28) at main.cc:956 > Is that correct? > > Regarding the former, generally at 20-30% CPU Bro shouldn't drop any > signficant amount of packets, there's no throttling mechanism or > such. One guess here would be the operating system. What kind of > system are you running on? Have you tried the tuning described on > http://www.net.t-labs.tu-berlin.de/research/bpcs/? I'm running FreeBSD 7.1 for i386. I had tried tuning based on the Bro Wiki, but the following page showed sysctl debug.bpf_bufsize and sysctl debug.bpf_maxbufsize. Those commands didn't work in FreeBSD 7.1. http://www.bro-ids.org/wiki/index.php/User_Manual:_Performance_Tuning The above tu-berlin.de link shows the following: sysctl -w net.bpf.bufsize=10485760 (10M) sysctl -w net.bpf.maxbufsize=10485760 (10M) The Bro-Workshop-July07-tierney.ppt showed the following should be added to the /etc/sysctl.conf net.bpf.bufsize=4194304 (4M) net.bpf.maxbufsize=8388608 (8M) Based on these two examples, I am guessing the bufsize is where the buffer starts, and the max is how large it can grow. Here are my default values: $sysctl -a |grep net.bpf net.bpf.maxbufsize: 524288 net.bpf.bufsize: 4096 According to the FreeBSD 7.1 manpage for sysctl, "The -w option has been deprecated and is silently ignored". I'll try setting both to 10M, like in the link you sent. $sysctl net.bpf.bufsize=10485760 $sysctl net.bpf.maxbufsize=10485760 I also added those values to the /etc/sysctl.conf so they get set on reboot. I just restarted the cluster, and the bro-1.4-robin process is sitting at 11-13%. The DroppedPackets/received ratio is flucuating between 3 and 25%. Shouldn't the CPU be maxing out before packets get dropped? > Another question: > is there any regularity in the timestamps of when the drops occur? > Like in regular intervals? (But longer intervals than 10s as that's > just the reporting interval). In the previous email, it looks like the intervals were 10s, but there was a gap of over a minute at epoch 1244425261.942659, which is right before the cluster froze. I'll try to keep an eye out for that if it happens again. > I wouldn't be totally surprised if the state checkpointing is the > culprit. To test that, can you remove the line "@load checkpoint" > from cluster.bro? I haven't tried this yet. I'll see if the bpf buffer increase helps. If not, I'll try unloading the checkpoint.bro. Tyler From robin at icir.org Fri Jun 12 16:02:40 2009 From: robin at icir.org (Robin Sommer) Date: Fri, 12 Jun 2009 16:02:40 -0700 Subject: [Bro] Stand-alone cluster problems In-Reply-To: <4A3291ED.2070807@colorado.edu> References: <4A2E945E.4070704@colorado.edu> <20090610172232.GA86099@icir.org> <4A3291ED.2070807@colorado.edu> Message-ID: <20090612230240.GE27493@icir.org> Hi Tyler, a number of thoughts: - increasing the BPF buffer sizes is definitly important and should generally reduce drops. - the top output shows that the Bro process is actually swapping out memory to disk: > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 51061 XXXXXX 1 -20 0 1207M 843M swread 1 606:53 0.00% Once that happens, pretty much all bets are off regarding drops: the I/O load will dominate everything else and Bro almost certainly drop tons of packets. - the stack backtrace shows a piece of code where we just found a problem in the version checked into my branch, which has the potential to cause drops with large tables. I can't tell whether it's a coincidence or indeed causing trouble in your case. In any case, I've committed a fix, please update your working copy and recompile to see whether that makes any difference. Robin -- Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From mcholste at gmail.com Tue Jun 16 07:50:07 2009 From: mcholste at gmail.com (Martin Holste) Date: Tue, 16 Jun 2009 09:50:07 -0500 Subject: [Bro] Hardware Experience In-Reply-To: <367A762AC7E2D6418E427F1490173F1703E9AE29@EXVBE014-11.exch014.msoutlookonline.net> References: <367A762AC7E2D6418E427F1490173F1703E9AE29@EXVBE014-11.exch014.msoutlookonline.net> Message-ID: Cool! But I can't believe you're Bro instance is doing much inspecting if it's receiving line-rate packets and only using 1% CPU. As I said before, the majority of the CPU time is usually in pattern matching and protocol decoding (which is basically pattern matching), so I'm assuming that unless the pattern matching is also hardware accelerated, you're not pattern matching much of the traffic being sent to Bro. Is that the case? Thanks, Martin On Tue, Jun 16, 2009 at 9:34 AM, Jens Christophersen wrote: > Hi Jason and Martin, > > > > I have with interest read mail tread about Napatech NT20E adapters. > > > > The NT20E adapter is able to capture data at line speed for any frame size > from 64 bytes to 10000 bytes without slicing the frames. The NT20E support > many forms of slicing so the NT20E adapter can be setup to slice frames if > you want to reduce the amount of data transferred to the server memory, but > for a ?Bro? application you probably don?t want to slice frames. > > > > If you want high ?Bro? performance I can recommend that you setup the NT20E > to distribute frames to the number of CPU cores in your server (e.g. 8) > based on 5-tuple hash key. When you are using the Napatech zero-copy LibPCAP > you start the Napatech LibPcap library with a command file with the > following commands: > > DeleteFilter = All > > SetupPacketFeedEngine[ TimeStampFormat=PCAP; > > DescriptorType=PCAP; > > MaxLatency=1000; > > SegmentSize=4096; > > Numfeeds=8 ] > > PacketFeedCreate[ NumSegments=128; Feed=(0..7) ] > > HashMode = Hash5TupleSorted > > Capture[ Feed = (0..7) ] = All > > > > Then frames are distributed to the 8 CPUs with a server CPU utilization of > less than 1% at full network load, so you have the full server CPU for your > Bro application. > > > > Best regards, Jens > > > *Yours Sincerely*** > > *Jens Christophersen*** > > *Chief Technology Officer* > > > > *Napatech A/S* > > Tobaksvejen 23A Phone: +45 4596 1500 > > DK-2860 S?borg Fax: +45 6980 2970 > > Denmark Mobile: +45 3091 5773 > > www.napatech.com E-mail: jc at napatech.com > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20090616/a635b670/attachment.html From rreitz at fnal.gov Tue Jun 16 08:36:29 2009 From: rreitz at fnal.gov (Randolph Reitz) Date: Tue, 16 Jun 2009 10:36:29 -0500 Subject: [Bro] svn error updating source Message-ID: <53E96ED2-9325-4D64-BB25-0299A6CB12DA@fnal.gov> While trying to update the tree... [brother at gumshoe ~/work]$ svn info Path: . URL: http://svn.icir.org/bro/branches/robin/work Repository Root: http://svn.icir.org/bro Repository UUID: 040645db-9414-0410-b69e-f32faa466a09 Revision: 6761 Node Kind: directory Schedule: normal Last Changed Author: robin Last Changed Rev: 6702 Last Changed Date: 2009-05-07 16:06:55 -0500 (Thu, 07 May 2009) I get this error... svn: Server sent unexpected return value (403 Forbidden) in response to OPTIONS request for 'http://svn.icir.org/bro' When I re-run 'svn update', more files are update and then the same error appears. Thanks, Randy From Tyler.Schoenke at colorado.edu Tue Jun 16 09:40:35 2009 From: Tyler.Schoenke at colorado.edu (Tyler T. Schoenke) Date: Tue, 16 Jun 2009 10:40:35 -0600 Subject: [Bro] Stand-alone cluster problems In-Reply-To: <20090612230240.GE27493@icir.org> References: <4A2E945E.4070704@colorado.edu> <20090610172232.GA86099@icir.org> <4A3291ED.2070807@colorado.edu> <20090612230240.GE27493@icir.org> Message-ID: <4A37CB03.80801@colorado.edu> Robin, I pulled the svn code from 6/15, and got the following error when running make: Generating code for SSLRecordLayerFlow make all-am g++ -DHAVE_CONFIG_H -I. -I.. -I. -I../aux/binpac/lib -I../src -I. -I.. -O -I/usr/local/include -W -Wall -Wno-unused -g -O2 -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.cc In file included from Frame.h:11, from main.cc:31: Val.h:853: error: extra qualification 'TableVal::' on member 'RecursiveSize' main.cc: In function 'int main(int, char**)': main.cc:452: warning: deprecated conversion from string constant to 'char*' *** Error code 1 Am I missing some dependency? In the meantime, I'll try unloading some of the scripts to drop the memory usage. Thanks, Tyler Robin Sommer wrote: > Hi Tyler, > > a number of thoughts: > > - increasing the BPF buffer sizes is definitly important and should > generally reduce drops. > > - the top output shows that the Bro process is actually swapping out > memory to disk: > >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND >> 51061 XXXXXX 1 -20 0 1207M 843M swread 1 606:53 0.00% > > Once that happens, pretty much all bets are off regarding drops: the > I/O load will dominate everything else and Bro almost certainly drop > tons of packets. > > - the stack backtrace shows a piece of code where we just found a > problem in the version checked into my branch, which has the > potential to cause drops with large tables. I can't tell whether > it's a coincidence or indeed causing trouble in your case. In any > case, I've committed a fix, please update your working copy and > recompile to see whether that makes any difference. > > Robin > From jchambers at ucla.edu Tue Jun 16 14:58:07 2009 From: jchambers at ucla.edu (Jason Chambers) Date: Tue, 16 Jun 2009 14:58:07 -0700 Subject: [Bro] Hardware Experience In-Reply-To: References: <367A762AC7E2D6418E427F1490173F1703E9AE29@EXVBE014-11.exch014.msoutlookonline.net> Message-ID: <4A38156F.7000006@ucla.edu> I believe the 1% utilization is in reference to the slight CPU interaction with DMA transfers. The rest is left to . >From what I understand of the documentation there are some pattern matching capabilities in the NT20E that might be leveraged to deliver specific traffic to the application. I'm still reading if and how running multiple Bro instances (using cpuset [1]) dedicated to certain types of traffic will work. [1] http://www.freebsd.org/cgi/man.cgi?query=cpuset&sektion=1&apropos=0&manpath=FreeBSD+7.2-RELEASE HTH, --Jason Martin Holste wrote: > Cool! But I can't believe you're Bro instance is doing much inspecting > if it's receiving line-rate packets and only using 1% CPU. As I said > before, the majority of the CPU time is usually in pattern matching and > protocol decoding (which is basically pattern matching), so I'm assuming > that unless the pattern matching is also hardware accelerated, you're > not pattern matching much of the traffic being sent to Bro. Is that the > case? > > Thanks, > > Martin > > On Tue, Jun 16, 2009 at 9:34 AM, Jens Christophersen > wrote: > > Hi Jason and Martin, > > > > I have with interest read mail tread about Napatech NT20E adapters. > > > > The NT20E adapter is able to capture data at line speed for any > frame size from 64 bytes to 10000 bytes without slicing the frames. > The NT20E support many forms of slicing so the NT20E adapter can be > setup to slice frames if you want to reduce the amount of data > transferred to the server memory, but for a ?Bro? application you > probably don?t want to slice frames. > > > > If you want high ?Bro? performance I can recommend that you setup > the NT20E to distribute frames to the number of CPU cores in your > server (e.g. 8) based on 5-tuple hash key. When you are using the > Napatech zero-copy LibPCAP you start the Napatech LibPcap library > with a command file with the following commands: > > DeleteFilter = All > > SetupPacketFeedEngine[ TimeStampFormat=PCAP; > > DescriptorType=PCAP; > > MaxLatency=1000; > > SegmentSize=4096; > > Numfeeds=8 ] > > PacketFeedCreate[ NumSegments=128; Feed=(0..7) ] > > HashMode = Hash5TupleSorted > > Capture[ Feed = (0..7) ] = All > > > > Then frames are distributed to the 8 CPUs with a server CPU > utilization of less than 1% at full network load, so you have the > full server CPU for your Bro application. > > > > Best regards, Jens > > > > > *Yours Sincerely*** > > *Jens Christophersen*** > > *Chief Technology Officer* > > > > *Napatech A/S* > > Tobaksvejen 23A Phone: +45 4596 1500 > > DK-2860 S?borg Fax: +45 6980 2970 > > Denmark Mobile: +45 3091 5773 > > www.napatech.com E-mail: > jc at napatech.com > > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From robin at icir.org Tue Jun 16 20:52:27 2009 From: robin at icir.org (Robin Sommer) Date: Tue, 16 Jun 2009 20:52:27 -0700 Subject: [Bro] Stand-alone cluster problems In-Reply-To: <4A37CB03.80801@colorado.edu> References: <4A2E945E.4070704@colorado.edu> <20090610172232.GA86099@icir.org> <4A3291ED.2070807@colorado.edu> <20090612230240.GE27493@icir.org> <4A37CB03.80801@colorado.edu> Message-ID: <20090617035227.GA77561@icir.org> On Tue, Jun 16, 2009 at 10:40 -0600, Tyler T. Schoenke wrote: > Val.h:853: error: extra qualification 'TableVal::' on member 'RecursiveSize' That got introduced by the recent merge and dependennt on the gcc version it does or does not generate an error. Now fixed. Robin -- Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Tue Jun 16 20:57:24 2009 From: robin at icir.org (Robin Sommer) Date: Tue, 16 Jun 2009 20:57:24 -0700 Subject: [Bro] svn error updating source In-Reply-To: <53E96ED2-9325-4D64-BB25-0299A6CB12DA@fnal.gov> References: <53E96ED2-9325-4D64-BB25-0299A6CB12DA@fnal.gov> Message-ID: <20090617035724.GB77561@icir.org> On Tue, Jun 16, 2009 at 10:36 -0500, Randolph Reitz wrote: > svn: Server sent unexpected return value (403 Forbidden) in response > to OPTIONS request for 'http://svn.icir.org/bro' Hmmm... Strange. I just tried a fresh checkout and that worked fine. Can you try that? Anyone else seeing this problem? Robin -- Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From rreitz at fnal.gov Wed Jun 17 19:26:21 2009 From: rreitz at fnal.gov (Randolph Reitz) Date: Wed, 17 Jun 2009 21:26:21 -0500 Subject: [Bro] svn error updating source In-Reply-To: <20090617035724.GB77561@icir.org> References: <53E96ED2-9325-4D64-BB25-0299A6CB12DA@fnal.gov> <20090617035724.GB77561@icir.org> Message-ID: <8DCAEA4F-355F-437B-AD38-266E18FCED98@fnal.gov> Checkout wroked... [brother at gumshoe ~]$ svn checkout http://svn.icir.org/bro/branches/robin/work A work/aux/broccoli/docs/html/c84.html A work/aux/broccoli/docs/html/a3617.html A work/aux/broccoli/docs/html/c20.html A work/aux/broccoli/docs/html/c54.html U work/aux/broccoli/docs/broccoli-manual.sgml Skipped 'work/policy/scan.bro' U work/policy/ftp.bro U work/policy/bro.init U work/policy/mime.bro U work/policy/http-request.bro U work/policy/all.bro U work/policy/sigs/dpd.sig U work/policy/dns.bro A work/policy/http-detect-passwd.bro A work/policy/capture-loss.bro U work Checked out revision 6765. Thanks, Randy On Jun 16, 2009, at 10:57 PM, Robin Sommer wrote: > > On Tue, Jun 16, 2009 at 10:36 -0500, Randolph Reitz wrote: > >> svn: Server sent unexpected return value (403 Forbidden) in response >> to OPTIONS request for 'http://svn.icir.org/bro' > > Hmmm... Strange. I just tried a fresh checkout and that worked > fine. Can you try that? Anyone else seeing this problem? > > Robin > > -- > Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org > ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jones at tacc.utexas.edu Fri Jun 19 10:50:46 2009 From: jones at tacc.utexas.edu (William L. Jones) Date: Fri, 19 Jun 2009 12:50:46 -0500 Subject: [Bro] Just installed bro cluster In-Reply-To: <2219B351-FDA0-445F-8FBE-144C537F83F4@osu.edu> References: <2219B351-FDA0-445F-8FBE-144C537F83F4@osu.edu> Message-ID: <0E07074B82CE4B4A9982802A8484B6968364AD426A@EXCHANGE2K7.tacc.utexas.edu> The worker node is on an amd system with 4 cores running linux, it is just a starter system made from spare part and will be expanded at some future date to a full bro cluster with maniple work machines. It has two dual port 10 GigE interfaces. Here are few thing that I think need a little work in the bro cluster setup. * The interface definition needs to allow multiple interfaces to be specified. In my the output of a tap is feed in a dual port 10 GigE card so I have to have bro read from two network interfaces. Right now you can work around the problem by just adding -I on the node interface configuration line but I think it deserves a more formal solution. * One of my 10 GigE circuits has 3 vlans on it which show up as 6 interfaces. With the non cluster version of bro I just ran with 3 different configuration files and kept logs and reports in three separate directories. One important side affect was that it allowed 3 separate instances of bro so that the system could spread the load across multiple cpus instead of one. With bro cluster I could not run 3 bro works on one machine due to the way the works and server talk to each other. I think it would be an important enhancements to bro cluster to allow multiple bro work instances on the same machine. Bill Jones From hall.692 at osu.edu Fri Jun 19 11:16:43 2009 From: hall.692 at osu.edu (Seth Hall) Date: Fri, 19 Jun 2009 14:16:43 -0400 Subject: [Bro] Just installed bro cluster In-Reply-To: <0E07074B82CE4B4A9982802A8484B6968364AD426A@EXCHANGE2K7.tacc.utexas.edu> References: <2219B351-FDA0-445F-8FBE-144C537F83F4@osu.edu> <0E07074B82CE4B4A9982802A8484B6968364AD426A@EXCHANGE2K7.tacc.utexas.edu> Message-ID: <16891353-08B8-4BFA-9B0D-E2951D430844@osu.edu> On Jun 19, 2009, at 1:50 PM, William L. Jones wrote: > * The interface definition needs to allow multiple interfaces to be > specified. In my the output of a tap is feed in a dual port 10 GigE > card so I have to have bro read from two network interfaces. Right > now you can work around the problem by just adding -I interface> on the node interface configuration line but I think it > deserves a more formal solution. I've thought about asking Robin to add this feature too, but it's probably better to bind your interfaces together at a lower level. The easy way of doing it is to create a bridge and bind all of the interfaces you want to sniff attached to the bridge. Then you only need to define a single interface in your Bro configuration. You could also do it with the netgraph subsystem FreeBSD too, but that's a bit more complicated. > * One of my 10 GigE circuits has 3 vlans on it which show up as 6 > interfaces. With the non cluster version of bro I just ran with 3 > different configuration files and kept logs and reports in three > separate directories. One important side affect was that it allowed > 3 separate instances of bro so that the system could spread the load > across multiple cpus instead of one. With bro cluster I could not > run 3 bro works on one machine due to the way the works and server > talk to each other. I think it would be an important enhancements > to bro cluster to allow multiple bro work instances on the same > machine. That already works. In your node.cfg file, you just need to define multiple workers and give them the same host option, but give them each their own interface to sniff. I'm using this on my cluster now to help make better use of the available cores until the multi-core work is functional. .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From jones at tacc.utexas.edu Fri Jun 19 12:05:33 2009 From: jones at tacc.utexas.edu (William L. Jones) Date: Fri, 19 Jun 2009 14:05:33 -0500 Subject: [Bro] Just installed bro cluster In-Reply-To: <16891353-08B8-4BFA-9B0D-E2951D430844@osu.edu> References: <2219B351-FDA0-445F-8FBE-144C537F83F4@osu.edu> <0E07074B82CE4B4A9982802A8484B6968364AD426A@EXCHANGE2K7.tacc.utexas.edu> <16891353-08B8-4BFA-9B0D-E2951D430844@osu.edu> Message-ID: <0E07074B82CE4B4A9982802A8484B6968364AD428E@EXCHANGE2K7.tacc.utexas.edu> Thanks your node configuration suggestion did work. I am not sure what my earlier problem was, probably just an a typo. Bill Jones -----Original Message----- From: Seth Hall [mailto:hall.692 at osu.edu] Sent: Friday, June 19, 2009 1:17 PM To: William L. Jones Cc: Bro List Subject: Re: [Bro] Just installed bro cluster On Jun 19, 2009, at 1:50 PM, William L. Jones wrote: > * The interface definition needs to allow multiple interfaces to be > specified. In my the output of a tap is feed in a dual port 10 GigE > card so I have to have bro read from two network interfaces. Right > now you can work around the problem by just adding -I interface> on the node interface configuration line but I think it > deserves a more formal solution. I've thought about asking Robin to add this feature too, but it's probably better to bind your interfaces together at a lower level. The easy way of doing it is to create a bridge and bind all of the interfaces you want to sniff attached to the bridge. Then you only need to define a single interface in your Bro configuration. You could also do it with the netgraph subsystem FreeBSD too, but that's a bit more complicated. > * One of my 10 GigE circuits has 3 vlans on it which show up as 6 > interfaces. With the non cluster version of bro I just ran with 3 > different configuration files and kept logs and reports in three > separate directories. One important side affect was that it allowed > 3 separate instances of bro so that the system could spread the load > across multiple cpus instead of one. With bro cluster I could not > run 3 bro works on one machine due to the way the works and server > talk to each other. I think it would be an important enhancements > to bro cluster to allow multiple bro work instances on the same > machine. That already works. In your node.cfg file, you just need to define multiple workers and give them the same host option, but give them each their own interface to sniff. I'm using this on my cluster now to help make better use of the available cores until the multi-core work is functional. .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From jones at tacc.utexas.edu Fri Jun 19 12:18:18 2009 From: jones at tacc.utexas.edu (William L. Jones) Date: Fri, 19 Jun 2009 14:18:18 -0500 Subject: [Bro] Just installed bro cluster In-Reply-To: <16891353-08B8-4BFA-9B0D-E2951D430844@osu.edu> References: <2219B351-FDA0-445F-8FBE-144C537F83F4@osu.edu> <0E07074B82CE4B4A9982802A8484B6968364AD426A@EXCHANGE2K7.tacc.utexas.edu> <16891353-08B8-4BFA-9B0D-E2951D430844@osu.edu> Message-ID: <0E07074B82CE4B4A9982802A8484B6968364AD4293@EXCHANGE2K7.tacc.utexas.edu> Found my earlier problem. If a name the works "work1,work2,..,worknn" the node configuration works. If I name the works "I2,NLR and internet" I run into my communication problem. Bill Jones -----Original Message----- From: Seth Hall [mailto:hall.692 at osu.edu] Sent: Friday, June 19, 2009 1:17 PM To: William L. Jones Cc: Bro List Subject: Re: [Bro] Just installed bro cluster On Jun 19, 2009, at 1:50 PM, William L. Jones wrote: > * The interface definition needs to allow multiple interfaces to be > specified. In my the output of a tap is feed in a dual port 10 GigE > card so I have to have bro read from two network interfaces. Right > now you can work around the problem by just adding -I interface> on the node interface configuration line but I think it > deserves a more formal solution. I've thought about asking Robin to add this feature too, but it's probably better to bind your interfaces together at a lower level. The easy way of doing it is to create a bridge and bind all of the interfaces you want to sniff attached to the bridge. Then you only need to define a single interface in your Bro configuration. You could also do it with the netgraph subsystem FreeBSD too, but that's a bit more complicated. > * One of my 10 GigE circuits has 3 vlans on it which show up as 6 > interfaces. With the non cluster version of bro I just ran with 3 > different configuration files and kept logs and reports in three > separate directories. One important side affect was that it allowed > 3 separate instances of bro so that the system could spread the load > across multiple cpus instead of one. With bro cluster I could not > run 3 bro works on one machine due to the way the works and server > talk to each other. I think it would be an important enhancements > to bro cluster to allow multiple bro work instances on the same > machine. That already works. In your node.cfg file, you just need to define multiple workers and give them the same host option, but give them each their own interface to sniff. I'm using this on my cluster now to help make better use of the available cores until the multi-core work is functional. .Seth --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From hall.692 at osu.edu Fri Jun 19 12:36:00 2009 From: hall.692 at osu.edu (Seth Hall) Date: Fri, 19 Jun 2009 15:36:00 -0400 Subject: [Bro] Just installed bro cluster In-Reply-To: <0E07074B82CE4B4A9982802A8484B6968364AD4293@EXCHANGE2K7.tacc.utexas.edu> References: <2219B351-FDA0-445F-8FBE-144C537F83F4@osu.edu> <0E07074B82CE4B4A9982802A8484B6968364AD426A@EXCHANGE2K7.tacc.utexas.edu> <16891353-08B8-4BFA-9B0D-E2951D430844@osu.edu> <0E07074B82CE4B4A9982802A8484B6968364AD4293@EXCHANGE2K7.tacc.utexas.edu> Message-ID: Interesting. If you can make it break reliably, please file a ticket at http://tracker.icir.org/bro Thanks! .Seth On Jun 19, 2009, at 3:18 PM, William L. Jones wrote: > Found my earlier problem. If a name the works > "work1,work2,..,worknn" the node configuration works. If I name the > works "I2,NLR and internet" I run into my communication problem. > > Bill Jones > > -----Original Message----- > From: Seth Hall [mailto:hall.692 at osu.edu] > Sent: Friday, June 19, 2009 1:17 PM > To: William L. Jones > Cc: Bro List > Subject: Re: [Bro] Just installed bro cluster > > > On Jun 19, 2009, at 1:50 PM, William L. Jones wrote: > >> * The interface definition needs to allow multiple interfaces to be >> specified. In my the output of a tap is feed in a dual port 10 GigE >> card so I have to have bro read from two network interfaces. Right >> now you can work around the problem by just adding -I > interface> on the node interface configuration line but I think it >> deserves a more formal solution. > > I've thought about asking Robin to add this feature too, but it's > probably better to bind your interfaces together at a lower level. > The easy way of doing it is to create a bridge and bind all of the > interfaces you want to sniff attached to the bridge. Then you only > need to define a single interface in your Bro configuration. You > could also do it with the netgraph subsystem FreeBSD too, but that's a > bit more complicated. > >> * One of my 10 GigE circuits has 3 vlans on it which show up as 6 >> interfaces. With the non cluster version of bro I just ran with 3 >> different configuration files and kept logs and reports in three >> separate directories. One important side affect was that it allowed >> 3 separate instances of bro so that the system could spread the load >> across multiple cpus instead of one. With bro cluster I could not >> run 3 bro works on one machine due to the way the works and server >> talk to each other. I think it would be an important enhancements >> to bro cluster to allow multiple bro work instances on the same >> machine. > > That already works. In your node.cfg file, you just need to define > multiple workers and give them the same host option, but give them > each their own interface to sniff. I'm using this on my cluster now > to help make better use of the available cores until the multi-core > work is functional. > > .Seth > > --- > Seth Hall > Network Security - Office of the CIO > The Ohio State University > Phone: 614-292-9721 > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro --- Seth Hall Network Security - Office of the CIO The Ohio State University Phone: 614-292-9721 From robin at icir.org Mon Jun 22 00:26:15 2009 From: robin at icir.org (Robin Sommer) Date: Mon, 22 Jun 2009 00:26:15 -0700 Subject: [Bro] Just installed bro cluster In-Reply-To: <0E07074B82CE4B4A9982802A8484B6968364AD426A@EXCHANGE2K7.tacc.utexas.edu> References: <2219B351-FDA0-445F-8FBE-144C537F83F4@osu.edu> <0E07074B82CE4B4A9982802A8484B6968364AD426A@EXCHANGE2K7.tacc.utexas.edu> Message-ID: <20090622072615.GA19003@icir.org> On Fri, Jun 19, 2009 at 12:50 -0500, you wrote: > * The interface definition needs to allow multiple interfaces to be > specified. I There's already a feature request filed for this in the tracker. I'm planning to add it soon. > With bro cluster I could not run 3 bro works on one machine due to the way the works and server talk to each other. As Seth already wrote, this is supposed to work. Please file a ticket with the details of the effect you're seeing. Robin -- Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From msz at seclab.tuwien.ac.at Mon Jun 22 18:26:09 2009 From: msz at seclab.tuwien.ac.at (Martin Szydlowski) Date: Mon, 22 Jun 2009 18:26:09 -0700 Subject: [Bro] Patch for BitTorrent analyzer Message-ID: <4A402F31.50008@seclab.tuwien.ac.at> Hi Bro Team, This is a patch that extends the functionality of the BitTorrent analyzer added by Nadi Sarrar and Bernd Ager. In particular, it will parse many popular extensions to the official protocol and also the azureus messaging protocol which uses a different message format. The patch has been thoroughly tested on off-line traces without causing problems. I am attaching the patch for both the 1.4 release and the latest svn revision (r6773) available and also a short description of the changes. greetz Martin -- Martin Szydlowski Vienna University of Technology (temporary @ UC Santa Barbara) Secure Systems Lab https://www.iseclab.org/ e-mail: msz at iseclab.org -------------- next part -------------- A non-text attachment was scrubbed... Name: bro-1.4-bittorrent.patch.gz Type: application/x-gzip Size: 5869 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20090622/4032a8b6/attachment.gz -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: bro-bittorrent.changes Url: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20090622/4032a8b6/attachment.ksh -------------- next part -------------- A non-text attachment was scrubbed... Name: bro-r6773-bittorrent.patch.gz Type: application/x-gzip Size: 5328 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20090622/4032a8b6/attachment-0001.gz -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20090622/4032a8b6/attachment.bin From ager at net.in.tum.de Tue Jun 23 03:53:49 2009 From: ager at net.in.tum.de (Bernhard Ager) Date: Tue, 23 Jun 2009 12:53:49 +0200 Subject: [Bro] Patch for BitTorrent analyzer In-Reply-To: <4A402F31.50008@seclab.tuwien.ac.at> References: <4A402F31.50008@seclab.tuwien.ac.at> Message-ID: <20090623105349.GF481@in.tum.de> Hi Martin, On Mon, Jun 22, 2009 at 06:26:09PM -0700, Martin Szydlowski wrote: > This is a patch that extends the functionality of the BitTorrent > analyzer added by Nadi Sarrar and Bernd Ager. In particular, it will > parse many popular extensions to the official protocol and also the > azureus messaging protocol which uses a different message format. The > patch has been thoroughly tested on off-line traces without causing > problems. I am attaching the patch for both the 1.4 release and the > latest svn revision (r6773) available and also a short description of > the changes. Oh, interesting to see that. Especially as I have also written a few extensions for the BitTorrent analyzer, among them the above mentioned Azureus Messaging Protocol (AZMP) and the LibTorrent Extension Protocol. My flavour of the enhanced parser analyzes the extension negotiation protocol (I remember that being a bit hairy) before deciding if AZMP or plain BitTorrent is used. Main differences between the analyzers as far as I can tell from a short peek into your code (forgive me if I mistook a thing or two): - AZMP is parsed generically, e.g., I don't handle AZ_PEER_EXCHANGE in a special way. When handling those messages explicitely, then this should be unified with LTEP anyway. - Well, LTEP and arbitraty message parsing integrated. - We use different methods for determining the transparent switch to the AZMP protocol. I can happily claim mine works correct for all examined connections---that includes a few TB of traffic :-) - I deliberately decided against changing the event signatures by adding an explicit PDU len field, though this is definitely the cleaner solution, and in case of padding it is the only way to communicate the correct PDU length. I have never seen a non-encrypted BT connection using padding in the wild, though. - Your analyzer handles a few non-standard message types that mine does only cover by the bittorrent_peer_unknown event (with no loss of information though). I've been testing my analyzer now on several TB of (live) BitTorrent traffic and it appears to run stable. For SSH subversion users all of this is available from svn+ssh://svn.icir.org/bro/branches/bager/bittorrent-enhancements but there appears to be no public access to this branch currently. Martin: If you are interested in the patch but don't have access, drop me a mail, then I'll send it to you. Bernhard -- Technische Universit?t Berlin An-Institut Deutsche Telekom Laboratories FG INET, Research Group Anja Feldmann Sekr. TEL 4 Ernst-Reuter-Platz 7 D-10587 Berlin From msz at seclab.tuwien.ac.at Tue Jun 23 11:29:03 2009 From: msz at seclab.tuwien.ac.at (Martin Szydlowski) Date: Tue, 23 Jun 2009 11:29:03 -0700 Subject: [Bro] Patch for BitTorrent analyzer In-Reply-To: <20090623105349.GF481@in.tum.de> References: <4A402F31.50008@seclab.tuwien.ac.at> <20090623105349.GF481@in.tum.de> Message-ID: <4A411EEF.5000204@seclab.tuwien.ac.at> Hi Bernhard, Bernhard Ager wrote: > Oh, interesting to see that. Especially as I have also written a few > extensions for the BitTorrent analyzer, among them the above mentioned > Azureus Messaging Protocol (AZMP) and the LibTorrent Extension > Protocol. I was not aware of that - I stopped following the SVN when 1.4 was released. Also, by that time, I had pretty much finished the extensions. It's been only a couple of fixes since then and my 1.4 patch still works fine on the most recent SVN trunk. I should have released it back then but somehow I kept forgetting, and only when I met Robin in Oakland last month I remembered. > My flavour of the enhanced parser analyzes the extension > negotiation protocol (I remember that being a bit hairy) before > deciding if AZMP or plain BitTorrent is used. I had planned to implement the negotiation protocol as well, but since I never encountered it in my traces I postponed that project :) > Main differences between the analyzers as far as I can tell from a > short peek into your code (forgive me if I mistook a thing or two): > > - AZMP is parsed generically, e.g., I don't handle AZ_PEER_EXCHANGE in > a special way. When handling those messages explicitely, then this > should be unified with LTEP anyway. > > - Well, LTEP and arbitraty message parsing integrated. > > - We use different methods for determining the transparent switch to > the AZMP protocol. I can happily claim mine works correct for all > examined connections---that includes a few TB of traffic :-) > > - I deliberately decided against changing the event signatures by > adding an explicit PDU len field, though this is definitely the > cleaner solution, and in case of padding it is the only way to > communicate the correct PDU length. I have never seen a > non-encrypted BT connection using padding in the wild, though. > > - Your analyzer handles a few non-standard message types that mine > does only cover by the bittorrent_peer_unknown event (with no loss > of information though). I have tailored the analyzer to my needs, and all the changes are a result of that, especially the PDU-len field and explicit recognition of certain extended message types (makes post-processing much easier for me). As you are the main developer for this analyzer and I but a mere tinkerer, I suggest you take whatever is useful from my patch and integrate it with your work. There is one actual bugfix in this patch for BitTorrentTracker.cc, that clears up an endiannes SNAFU when converting the peer list sent by the tracker to Bro host/port tuples. However, this is only interesting when using the extended bt-tracker-plist.bro policy that will also print the complete peer list to the log file (again something I need that others probably won't find usefull). > I've been testing my analyzer now on several TB of (live) BitTorrent > traffic and it appears to run stable. For SSH subversion users all of > this is available from > svn+ssh://svn.icir.org/bro/branches/bager/bittorrent-enhancements but > there appears to be no public access to this branch currently. Martin: > If you are interested in the patch but don't have access, drop me a > mail, then I'll send it to you. For now, I am content with what I have and need to focus on other things, but I would like to continue work on the BT analyzer when I have more time (couple of weeks from now). I'll let you know when that happens and if you are still working on BT/Bro then, I'd be delighted to cooperate. greetz Martin -- Martin Szydlowski Vienna University of Technology (temporary @ UC Santa Barbara) Secure Systems Lab https://www.iseclab.org/ e-mail: msz at iseclab.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20090623/c428c7af/attachment.bin From jones at tacc.utexas.edu Tue Jun 23 11:30:25 2009 From: jones at tacc.utexas.edu (William L. Jones) Date: Tue, 23 Jun 2009 13:30:25 -0500 Subject: [Bro] Is there any way to flush the conn log every so often Message-ID: <0E07074B82CE4B4A9982802A8484B6968364AD4461@EXCHANGE2K7.tacc.utexas.edu> I have a connection that I am monitoring with a low amount of activity so the conn log rarely fills up enough to be flushed to disk. I would like to force a flush on it every so often. Is there a way to do this through a bro config file. Bill Jones -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20090623/836d908c/attachment.html From robin at icir.org Tue Jun 23 13:50:18 2009 From: robin at icir.org (Robin Sommer) Date: Tue, 23 Jun 2009 13:50:18 -0700 Subject: [Bro] Is there any way to flush the conn log every so often In-Reply-To: <0E07074B82CE4B4A9982802A8484B6968364AD4461@EXCHANGE2K7.tacc.utexas.edu> References: <0E07074B82CE4B4A9982802A8484B6968364AD4461@EXCHANGE2K7.tacc.utexas.edu> Message-ID: <20090623205018.GB38237@icir.org> On Tue, Jun 23, 2009 at 13:30 -0500, you wrote: > I would like to force a flush on it every so often. Is there a way > to do this through a bro config file. Yes, there are two options: - file-flush.bro flushes all logs regurlarly (default: every 10s). - the built-in function set_buf() disables buffering for a particular log file; see the bro_init() handler in remote.bro for an example. If there's not much traffic on the line, disabling the buffering for conn.log shouldn't be a problem. Roin -- Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From jones at tacc.utexas.edu Thu Jun 25 12:09:24 2009 From: jones at tacc.utexas.edu (William L. Jones) Date: Thu, 25 Jun 2009 14:09:24 -0500 Subject: [Bro] Is there any way to flush the conn log every so often In-Reply-To: <20090623205018.GB38237@icir.org> References: <0E07074B82CE4B4A9982802A8484B6968364AD4461@EXCHANGE2K7.tacc.utexas.edu> <20090623205018.GB38237@icir.org> Message-ID: <0E07074B82CE4B4A9982802A8484B6968364AD4619@EXCHANGE2K7.tacc.utexas.edu> I used file-flush.bro to flush the logs. Thanks! -----Original Message----- From: Robin Sommer [mailto:robin at icir.org] Sent: Tuesday, June 23, 2009 3:50 PM To: William L. Jones Cc: bro at bro-ids.org Subject: Re: [Bro] Is there any way to flush the conn log every so often On Tue, Jun 23, 2009 at 13:30 -0500, you wrote: > I would like to force a flush on it every so often. Is there a way > to do this through a bro config file. Yes, there are two options: - file-flush.bro flushes all logs regurlarly (default: every 10s). - the built-in function set_buf() disables buffering for a particular log file; see the bro_init() handler in remote.bro for an example. If there's not much traffic on the line, disabling the buffering for conn.log shouldn't be a problem. Roin -- Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From jones at tacc.utexas.edu Thu Jun 25 12:18:51 2009 From: jones at tacc.utexas.edu (William L. Jones) Date: Thu, 25 Jun 2009 14:18:51 -0500 Subject: [Bro] Bro restart not saving logs properly In-Reply-To: <20090623205018.GB38237@icir.org> References: <0E07074B82CE4B4A9982802A8484B6968364AD4461@EXCHANGE2K7.tacc.utexas.edu> <20090623205018.GB38237@icir.org> Message-ID: <0E07074B82CE4B4A9982802A8484B6968364AD461F@EXCHANGE2K7.tacc.utexas.edu> I noticed that when I a restart bro using the cluster command that the current manager logs are not saved. What end up in the logs directive is data for only a few minutes as if bro reopened the logs before terminating. Could some verify that bro work properly under bsd. I am running bro under linux and suspect that this is only a problem under linux and not bsd. Bill Jones -----Original Message----- From: Robin Sommer [mailto:robin at icir.org] Sent: Tuesday, June 23, 2009 3:50 PM To: William L. Jones Cc: bro at bro-ids.org Subject: Re: [Bro] Is there any way to flush the conn log every so often On Tue, Jun 23, 2009 at 13:30 -0500, you wrote: > I would like to force a flush on it every so often. Is there a way > to do this through a bro config file. Yes, there are two options: - file-flush.bro flushes all logs regurlarly (default: every 10s). - the built-in function set_buf() disables buffering for a particular log file; see the bro_init() handler in remote.bro for an example. If there's not much traffic on the line, disabling the buffering for conn.log shouldn't be a problem. Roin -- Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From pudding33 at gmail.com Thu Jun 25 22:56:21 2009 From: pudding33 at gmail.com (shan shan) Date: Fri, 26 Jun 2009 14:56:21 +0900 Subject: [Bro] Fwd: Some Questions, please In-Reply-To: <3ab9914e0906252246r74c337ado5ba6663fd86e2bce@mail.gmail.com> References: <3ab9914e0906252246r74c337ado5ba6663fd86e2bce@mail.gmail.com> Message-ID: <3ab9914e0906252256w216193dcxe7af3797460de3b3@mail.gmail.com> Good Afternoon.My questions are as follows: Q1. I can not solve the problem when doing bro_config, and the command lines are in the attached file config.txt . I do not know whether my configuration is setup right. Because I only get a log file in the /usr/local/bro/logs fold (in the attached file info.localhost.09-06-25_13.25.33). In the /usr/local/bro/reports folder there is no report file. Are the report generated automatically? Or shuld I generate it by hand? Q2. In the quick-Start file, I find that the report example. At the end of the report, there is a list of connections(only first 25 after alarm are listed). I want to ask: if there is no alarm, will there be no connections list (such as time and byte information)? And Bro can list only first 25 connections after alarm ? If I want the information of all connections, how can I get that? Thank you very much! I am looking forward for your reply.[?] -- Zhu Shan -- Zhu Shan -- Zhu Shan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20090626/9f737f3d/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 96 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20090626/9f737f3d/attachment.gif -------------- next part -------------- [root at localhost scripts]# ./bro_config Automode not enabled Running Bro Configuration Utility Configure settings in bro.cfg? (YES/no) [YES] Values enclosed in '[ ]'s are the default value set if you hit return. **** Detected previous bro.cfg file ***** May I use /usr/local/bro/etc/bro.cfg for defaults? [Y/n]Y Sourcing /usr/local/bro/etc/bro.cfg for defaults. Log archive directory [/usr/local/bro/archive] User id to install and run Bro under [root] Interface name to listen on. The default is to use the busiest one found. [eth0] Site name for reports (i.e. LBNL, FOO.COM, BAZ.ORG) [eth0] Starting time for a report run (0001 is 12:01 am and 1201 is 12:01pm) [0000] How often (in hours) to generate an activity report [24] Email reports? (YES/no) [YES] Email address for local reports to be mailed to [NO] Do you want to encrypt email reports (YES/NO) [NO] error: "net.core.rmem_max" must be of the form name=value error: Malformed setting "=" error: "16777216" must be of the form name=value ERROR: Can't change value, entry exists in /etc/sysctl.conf! May I guess your network configuration for you? [YES] Checking network Running localnets script to determine the local network range ... This will take about 20 seconds Capturing packets .... done. Analyzing dump file.....cannot execute ./adtrace/adtrace /tmp/bro_config.tcpdump.file.29010: ?????????????????????? done. grep: local.site.bro: ?????????????????????? ./bro_config: line 462: [: : integer expression expected Your network appears to contain the following networks: grep: local.site.bro: ?????????????????????? Edit local.site.bro by hand if this is not correct Bro Configuration Finished. Press any key to now to continue. You have new mail in /var/spool/mail/root [root at localhost scripts]# -------------- next part -------------- A non-text attachment was scrubbed... Name: info.localhost.09-06-25_13.25.33 Type: application/octet-stream Size: 2078 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20090626/9f737f3d/attachment.obj From buraglio at illinois.edu Mon Jun 29 14:22:05 2009 From: buraglio at illinois.edu (Nick Buraglio) Date: Mon, 29 Jun 2009 16:22:05 -0500 Subject: [Bro] Hardware Experience In-Reply-To: <20090528042100.GE12888@uiuc.edu> References: <8A6C3E52-76BD-46F5-B210-051D2D5392F4@illinois.edu> <4A1DA88F.1060603@ucar.edu> <20090528042100.GE12888@uiuc.edu> Message-ID: <61C42BB1-6CC7-4886-B123-83535186CF62@illinois.edu> I actually did quite a bit of the work with Aashish on the Dag and Myricom cards (I was the one that gave them to him back when I still worked at NCSA), and like he said we had lots of issues with them. Endace support was helpful but in the end it was a more supportable direction to go with the Intel and Myricom cards. Using NICs has proven to be very robust for us. I have the cards that I'd originally sent the mail out about running on a FreeBSD 7.2 system watching pretty heavily loaded links and so far have not seen any issues. nb --- Nick Buraglio Network Engineer, CITES, University of Illinois GPG key 0x2E5B44F4 Phone: 217.244.6428 buraglio at illinois.edu On May 27, 2009, at 11:21 PM, Aashish Sharma wrote: > Hi Sean: > > Back in 2006 we got 4 Dag 6.2SE cards to monitor our 10G links. > During the time we were running firmware 2.5.7.5. on the cards. We > had real hard time keeping Bro running reliably in a sustained > manner using Dag cards. We encountered a lot of issues - including > lack of drivers, lack of built in support for libpcap, crashing of > Bro repeatedly, heating up and crashing of system as well. > > In fact, Robin helped us quite a bit and even wrote drivers and > support for Dag in Bro. Endace support was prompt too and they > provided us with a new modified firmware but not much changed. > > During all that time, For production Bro we relied on a pair of > Intel 10G cards while we resolve this issue with Dag cards (spent > considerable time trying to get this working), > > All in all, we had lot of issues running Dag capture cards reliably. > Eventually, we gave up and got Myricom 10G cards. We have been > quite happy with Myricom cards and have not encountered any issues > since. > > Hope this helps, > > Aashish Sharma > NCSA > > > On Wed, May 27, 2009 at 02:54:39PM -0600, Sean McCreary wrote: >> I'd be careful about purchasing 10G NICs for packet capture. I >> have not >> been able to configure a FreeBSD 6.3 system with a Myricom Myri-10G >> NIC >> to reliably capture traffic on a lightly loaded link (~2Mb/s, ~240 >> kpps). One option I'm interested in trying is the Endace DAG, >> . Does >> anyone >> have experience using these cards with bro? >> >> Nick Buraglio wrote: >>> Good afternoon, list. I'm hoping to get a quick opinion on some >>> hardware. I've done some brief looking and not really found what >>> I'm >>> seeking so I'll post here in hopes that one of you can share some >>> experience. >>> I'm exploring deployment of some Bro boxes and was hoping to >>> leverage >>> a great deal that Sun is offering to get the hardware. I know that >>> the boxes can do what I need them to do, as I've worked on Bro >>> implementations elsewhere. What I'd really like to know is if >>> anyone >>> has used the Sun (Intel Chipset 82598) dual port 10g cards? >>> They're a >>> decent savings of capitol, but I'd rather just spend the money to >>> get >>> the cards I'm used to (single port 10g Intel or Myricom) if the dual >>> port cards behave strangely or are a time-vortex to get working. >>> I'm making an assumption that the dual port cards operate similar to >>> the single port cards. Has anyone used these in a bro deployment? >>> >>> >>> Thanks, >>> nb >>> --- >>> Nick Buraglio >>> Network Engineer, CITES, University of Illinois >>> GPG key 0x2E5B44F4 >>> Phone: 217.244.6428 >>> buraglio at illinois.edu >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From Terry.Barker at sas.com Tue Jun 30 09:49:54 2009 From: Terry.Barker at sas.com (Terry Barker) Date: Tue, 30 Jun 2009 12:49:54 -0400 Subject: [Bro] segmentation fault on first packet Message-ID: <6EE13C4BD5071F448CB1FD09CCF5C3520A6996961E@MERCMBX14.na.sas.com> I'm having trouble running bro 1.4 that I recently installed on a solaris computer (uname -a gives: SunOS fsm04 5.9 Generic_118558-39 sun4u sparc SUNW,Sun-Fire-V890). I've installed and run bro on linux boxes several times over the last couple of years and know the basics. The program core dumps on the first packet of several pcap files I?ve tried. For example, I tried bro on a pcap file used in a recent bro workshop tutorial called trace1.tcpdump, and I've attached the first 20 packets (in test.tcpdump) just to be sure we're on the same page. If I run bro -r test.tcpdump I get a segmentation fault on the first packet. This is the output from gdb ...... ________________________________________________________ warning: Temporarily disabling breakpoints for unloaded shared library "/usr/lib/ld.so.1" Program received signal SIGSEGV, Segmentation fault. ConnCompressor::PktHdrToPendingConn (this=0x494c9c, time=964800422.39454699, key=0x491788, ip=0x0, tp=0x397cf4, c=0x494c9c) at ConnCompressor.cc:617 617 c->time = time; (gdb) where #0 ConnCompressor::PktHdrToPendingConn (this=0x494c9c, time=964800422.39454699, key=0x491788, ip=0x0, tp=0x397cf4, c=0x494c9c) at ConnCompressor.cc:617 #1 0x000cd410 in ConnCompressor::FirstFromOrig (this=0x396e18, t=964800422.39454699, key=0x491788, ip=0xffbfe1c0, tp=0x397cf4) at ConnCompressor.cc:276 #2 0x000cdc7c in ConnCompressor::NextPacket (this=0x396e18, t=964800422.39454699, key=0x491788, ip=0xffbfe1c0, hdr=0x397760, pkt=0x397cd2 "") at ConnCompressor.cc:234 #3 0x001b6cb8 in NetSessions::DoNextPacket (this=0x3984f0, t=964800422.39454699, hdr=0x397760, ip_hdr=0xffbfe1c0, pkt=0x397cd2 "", hdr_size=14) at Sessions.cc:611 #4 0x001b73e8 in NetSessions::NextPacket (this=0x3984f0, t=964800422.39454699, hdr=0x397760, pkt=0x397cd2 "", hdr_size=14, pkt_elem=0x0) at Sessions.cc:305 #5 0x00176bf0 in net_packet_dispatch (t=964800422.39454699, hdr=0x397760, pkt=0x397cd2 "", hdr_size=14, src_ps=0x397728, pkt_elem=0x0) at Net.cc:434 #6 0x00176e54 in net_packet_arrival (t=964800422.39454699, hdr=0x397760, pkt=0x397cd2 "", hdr_size=14, src_ps=0x397728) at Net.cc:496 #7 0x001863a4 in PktSrc::Process (this=0x397728) at PktSrc.cc:199 #8 0x00177380 in net_run () at Net.cc:526 #9 0x0009c230 in main (argc=3550208, argv=0x362c00) at main.cc:977 ________________________________________________________ If I run bro with -t bro -t -r test.tcpdump this is the result: ________________________________________________________ Execution tracing ON. ./test.tcpdump, line 1: error: unrecognized character - ? ./test.tcpdump, line 1: error: unrecognized character - ? ./test.tcpdump, line 1: error: unrecognized character - ? ./test.tcpdump, line 1: error: unrecognized character - ? ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character -  ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character -  ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character -  ./test.tcpdump, line 1: error: unrecognized character - ? ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character -  ./test.tcpdump, line 1: error: unrecognized character - ? ./test.tcpdump, line 1: error: unrecognized character - ? ./test.tcpdump, line 1: error: unrecognized character - ? ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character -  ./test.tcpdump, line 1: error: unrecognized character - ? ./test.tcpdump, line 1: error: unrecognized character - ? ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character - @ ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character - @ ./test.tcpdump, line 1: error: unrecognized character -  ./test.tcpdump, line 1: error: unrecognized character - ? ./test.tcpdump, line 1: error: unrecognized character - ? ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character - ./test.tcpdump, line 1: error: unrecognized character -  ./test.tcpdump, line 1: error: unrecognized character -  ./test.tcpdump, line 1: error: unrecognized character - ? ./test.tcpdump, line 1: error: parse error, at or near "JA" ________________________________________________________ followed by the seg fault. Any advice would be appreciated. Terry -------------- next part -------------- A non-text attachment was scrubbed... Name: test.tcpdump Type: application/octet-stream Size: 1574 bytes Desc: test.tcpdump Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20090630/fd86f803/attachment.obj From vern at icir.org Tue Jun 30 09:57:38 2009 From: vern at icir.org (Vern Paxson) Date: Tue, 30 Jun 2009 09:57:38 -0700 Subject: [Bro] segmentation fault on first packet In-Reply-To: <6EE13C4BD5071F448CB1FD09CCF5C3520A6996961E@MERCMBX14.na.sas.com> (Tue, 30 Jun 2009 12:49:54 EDT). Message-ID: <200906301657.n5UGvhcX032485@pork.ICSI.Berkeley.EDU> I suspect this is due to an alignment issue with the connection compressor's tricks with pointers, but I'll let Robin comment on that. One immediate comment: bro -t -r test.tcpdump -t takes an argument, so this is creating an execution trace file called "-r" and then reading test.tcpdump as a Bro script, hence all of the "unrecognized character" complaints. Vern From ming-shih.wong at gd-ais.com Tue Jun 30 10:21:28 2009 From: ming-shih.wong at gd-ais.com (Wong, Ming-Shih) Date: Tue, 30 Jun 2009 12:21:28 -0500 Subject: [Bro] Bro IDS installation Message-ID: <91CC20228CD4E2408BF66B4C3C1201CD02222FFE@txsa01-mail01.ad.gd-ais.com> Dear Sir or Madam, We are new to Bro IDS and having some problem installing the system. Some of your documents are a bit out-dated. For example, "Bro Quick Start Guide" is dated 2004 for FreeBSD 4.10. Our box is installed with FreeBSD 7.2. We followed the guide to compile the source code, ./configure, make, make install and make install-brolite. We also manually executed the "bro_config" script to configure the system. But then executing "bro.rc start" failed and returned an error like "can't open .....". Could anyone provide some quick help? Thanks. Best regards, Mike Wong General Dynamics - Advanced Information Systems (210)442-4255 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20090630/f1331876/attachment.html From jones at tacc.utexas.edu Tue Jun 30 10:22:38 2009 From: jones at tacc.utexas.edu (William L. Jones) Date: Tue, 30 Jun 2009 12:22:38 -0500 Subject: [Bro] Hardware Experience In-Reply-To: <61C42BB1-6CC7-4886-B123-83535186CF62@illinois.edu> References: <8A6C3E52-76BD-46F5-B210-051D2D5392F4@illinois.edu> <4A1DA88F.1060603@ucar.edu> <20090528042100.GE12888@uiuc.edu> <61C42BB1-6CC7-4886-B123-83535186CF62@illinois.edu> Message-ID: <0E07074B82CE4B4A9982802A8484B6968364AD48DA@EXCHANGE2K7.tacc.utexas.edu> TACC is using the Sun dual port cards. The system runs bro cluster with ip filters to break the traffic up into multiple ip quadrants this allow different cpu to work on each quadrant of ip space. My rule of thumb is that it takes 1 cpu to process 1 Giga/bit of data. Right now the system is a 4 cpu system to monitor two 10 GigE connection, it just a starter system. I plan to upgraded it to two 8 cpu system each monitoring one 10 GigE connection later this year. I don't know how far this configuration will scale. Bill Jones -----Original Message----- From: bro-bounces at ICSI.Berkeley.EDU [mailto:bro-bounces at ICSI.Berkeley.EDU] On Behalf Of Nick Buraglio Sent: Monday, June 29, 2009 4:22 PM To: bro at bro-ids.org Subject: Re: [Bro] Hardware Experience I actually did quite a bit of the work with Aashish on the Dag and Myricom cards (I was the one that gave them to him back when I still worked at NCSA), and like he said we had lots of issues with them. Endace support was helpful but in the end it was a more supportable direction to go with the Intel and Myricom cards. Using NICs has proven to be very robust for us. I have the cards that I'd originally sent the mail out about running on a FreeBSD 7.2 system watching pretty heavily loaded links and so far have not seen any issues. nb --- Nick Buraglio Network Engineer, CITES, University of Illinois GPG key 0x2E5B44F4 Phone: 217.244.6428 buraglio at illinois.edu On May 27, 2009, at 11:21 PM, Aashish Sharma wrote: > Hi Sean: > > Back in 2006 we got 4 Dag 6.2SE cards to monitor our 10G links. > During the time we were running firmware 2.5.7.5. on the cards. We > had real hard time keeping Bro running reliably in a sustained > manner using Dag cards. We encountered a lot of issues - including > lack of drivers, lack of built in support for libpcap, crashing of > Bro repeatedly, heating up and crashing of system as well. > > In fact, Robin helped us quite a bit and even wrote drivers and > support for Dag in Bro. Endace support was prompt too and they > provided us with a new modified firmware but not much changed. > > During all that time, For production Bro we relied on a pair of > Intel 10G cards while we resolve this issue with Dag cards (spent > considerable time trying to get this working), > > All in all, we had lot of issues running Dag capture cards reliably. > Eventually, we gave up and got Myricom 10G cards. We have been > quite happy with Myricom cards and have not encountered any issues > since. > > Hope this helps, > > Aashish Sharma > NCSA > > > On Wed, May 27, 2009 at 02:54:39PM -0600, Sean McCreary wrote: >> I'd be careful about purchasing 10G NICs for packet capture. I >> have not >> been able to configure a FreeBSD 6.3 system with a Myricom Myri-10G >> NIC >> to reliably capture traffic on a lightly loaded link (~2Mb/s, ~240 >> kpps). One option I'm interested in trying is the Endace DAG, >> . Does >> anyone >> have experience using these cards with bro? >> >> Nick Buraglio wrote: >>> Good afternoon, list. I'm hoping to get a quick opinion on some >>> hardware. I've done some brief looking and not really found what >>> I'm >>> seeking so I'll post here in hopes that one of you can share some >>> experience. >>> I'm exploring deployment of some Bro boxes and was hoping to >>> leverage >>> a great deal that Sun is offering to get the hardware. I know that >>> the boxes can do what I need them to do, as I've worked on Bro >>> implementations elsewhere. What I'd really like to know is if >>> anyone >>> has used the Sun (Intel Chipset 82598) dual port 10g cards? >>> They're a >>> decent savings of capitol, but I'd rather just spend the money to >>> get >>> the cards I'm used to (single port 10g Intel or Myricom) if the dual >>> port cards behave strangely or are a time-vortex to get working. >>> I'm making an assumption that the dual port cards operate similar to >>> the single port cards. Has anyone used these in a bro deployment? >>> >>> >>> Thanks, >>> nb >>> --- >>> Nick Buraglio >>> Network Engineer, CITES, University of Illinois >>> GPG key 0x2E5B44F4 >>> Phone: 217.244.6428 >>> buraglio at illinois.edu >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From robin at icir.org Tue Jun 30 13:39:06 2009 From: robin at icir.org (Robin Sommer) Date: Tue, 30 Jun 2009 13:39:06 -0700 Subject: [Bro] segmentation fault on first packet In-Reply-To: <200906301657.n5UGvhcX032485@pork.ICSI.Berkeley.EDU> References: <6EE13C4BD5071F448CB1FD09CCF5C3520A6996961E@MERCMBX14.na.sas.com> <200906301657.n5UGvhcX032485@pork.ICSI.Berkeley.EDU> Message-ID: <20090630203906.GF21883@icir.org> On Tue, Jun 30, 2009 at 09:57 -0700, Vern Paxson wrote: > I suspect this is due to an alignment issue with the connection compressor's > tricks with pointers, but I'll let Robin comment on that. I suspect so too. Terry, can you try running with an additional "use_connection_compressor=F" on the command line and see if that avoids the crash? Robin -- Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org