[Bro] Dropping packets - How do I leverage multiple core with BRO?

[Tyler T. Schoenke]

I forgot that I already put it on the Wiki.

http://www.bro-ids.org/wiki/index.php/ClusterFrontendClickModularRouter

Tyler

On 11/12/2010 09:01 AM, Veronica Estrada wrote:
> Thank you for your detailed answer.  I am doing analysis on terabytes
> of data that is why i need load balance.
>
> Could you send me the config?
> Veronica
>
>
> On Fri, Nov 12, 2010 at 2:11 AM, Tyler T. Schoenke
> <Tyler.Schoenke at colorado.edu> wrote:
>   
>> This may be possible.  I just Googled and saw there is a program called
>> tcprelay that can be used to feed a pcap into an Ethernet interface.
>> You could use tcprelay to feed the pcap into the Click! Modular Router
>> and have Click! load balance the traffic to a Bro cluster with many
>> workers to utilize all your cores.
>>
>> The cluster is quite easy to set up, and the Click! interface config is
>> pretty easy as well.  I have a cluster of seven workers running on seven
>> of the cores in my server.  The eighth runs Click!.  The cluster manager
>> and proxy run on a recycled lab workstation with a big hard drive.
>>
>> If you are interested, I can send a copy of my Click! configuration.  It
>> is a modified version of Justin's that was posted to the list a while back.
>>
>> Using this type of setup, you could run the Bro manager, proxy(ies) as
>> well as 12 or 13 worker processes and Click! all on the same server.
>> The only reason I moved my manager and proxy off was to have more
>> workers processing traffic.  I think this will work with FreeBSD or
>> Linux.  Click! kernel mode requires Linux, but I don't think the load
>> balancing uses kernel drivers.
>>
>> You can grep the notice.log for Dropped to see how much traffic is not
>> being processed.  I don't recall the script that logs that, but it is
>> probably drop.bro.  I think it is on by default with the cluster config.
>>
>> Tyler
>>
>> --
>> Tyler Schoenke
>> Network Security Analyst
>> IT Security Office
>> University of Colorado - Boulder
>>
>>
>> On 11/11/2010 12:58 AM, Veronica Estrada wrote:
>>     
>>> Hello BRO professionals,
>>>
>>> I am using BRO v 1.5.1 to analyze off-line pcap files. When I run BRO
>>> on 4Gb pcap file, one CPU core always reaches 100% but the server
>>> still has more 15 idle cores.
>>> The analysis uses brolite, dpd and detect-protocols.
>>>
>>> I am afraid BRO is loosing packets. By the way, how can I measure
>>> packet dropping?
>>>
>>> The capture-loss generates this notice:
>>> no=CaptureLossSummary na=NOTICE_ALARM_ALWAYS msg=estimated\ rate\ \=\
>>> 0.0082201 tag=@36-6fb3-4a
>>>
>>> Are this events or bytes? WHy indicates tag? I cannot find any
>>> reference to this tag in any of the other logs. By reading the
>>> documentation, it seems you don't recommend this metric.
>>> Instead, I will be happy to know the number of packets that BRO
>>> processed. I cannot find where is this number logged.
>>>
>>> Best regards
>>>
>>> Veronica Estrada
>>> Nakao's Laboratory
>>> Univ. of Tokyo
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>
>>>
>>>       
>>