[Bro] To proxy or not to proxy...

[Adam Pumphrey]

Great discussion and pointers.  I’m working on a similar performance tuning and stabilization effort.  

I took a closer look to verify and I can confirm Aashish’s statement about the numbering of cores on Linux.  We’re running CENTOS 6.2.  This box has 2 hyperthreaded hex-core procs.   All physical cores are assigned sequential ID’s in socket/core order, then hyperthreaded cores are assigned sequential ID’s in socket/core order.   

Here’s what we end up with:

__socket0 (P/H)__
0/12
1/13
2/14
3/15
4/16
5/17

__socket1 (P/H)__
6/18
7/19
8/20
9/21
10/22
11/23

Adam


> On Apr 2, 2015, at 1:17 PM, Aashish Sharma <asharma at lbl.gov> wrote:
> 
> Same here: I have a proxy for every 10 workers on each of the physical box (which runs workers) in the cluster.
> 
> Ah! regarding CPU pinning: 
> 
>> fairly well. You may want to reduce your worker count a bit to leave 
>> enough CPUs for the proxies. Out of curiosity are you pinning your 
>> workers to dedicated CPU cores? If you are not it could be that your 
>> workers are bouncing between cores due to hyper-threading which can 
>> cause them to stomp all over each other. I found pinning workers to 
>> cores helped tremendously when it came to worker health.
> 
> I agree completely! 
> 
> Also, Make sure that you have enough cores to run workers on. 
> 
> With respect to CPU pinning, on *FreeBSD*, CPUs are numbered as :
> P = physical core 
> H = Hyperthread core
> 
> 0/1 = P/H 
> 2/3 = P/H 
> 4/5 = P/H 
> ...
> ...
> 11/12=P/H
> 
> You certainly don't want to pin_cpu on FreeBSD as 0,1,2,3 but instead pin_cpu=0,2,4,6,8.... (or 1,3,5,7...)
> 
> However, I beleive Linux does it different. While I have not yet looked at a Linux's box, I believe its scheme for hex-core processor is 
> 
> 0/6=P/H
> 1/7=P/H
> 2/8=P/H
> ..
> ..
> 5/12=P/H
> 
> so you might want to pin_cpu on linux as: pin_cpu=01,2,3,4,5 or (6,7,8,9,10,11,12)
> 
> Make sure you leave a few cores alone for proxy and other tasks when pinning. 
> 
> Oh, btw, we have found no noticible difference in performance at all, when you pin a bro process on only physical core vs only hyperthreded cores. But make sure you don't pin bro processes on both P/H at the same time.
> 
> Now, it would be great if someone can confirm the linux side of the story. or shed more light on cpu_pinning. 
> 
> 
> 
> 
> Aashish 
> 
> 
> On Thu, Apr 02, 2015 at 11:24:31AM -0500, Gary Faulkner wrote:
>> I'm currently running a separate box that has the manager and proxies on 
>> it, but I did just as you describe at one point and it seemed to work 
>> fairly well. You may want to reduce your worker count a bit to leave 
>> enough CPUs for the proxies. Out of curiosity are you pinning your 
>> workers to dedicated CPU cores? If you are not it could be that your 
>> workers are bouncing between cores due to hyper-threading which can 
>> cause them to stomp all over each other. I found pinning workers to 
>> cores helped tremendously when it came to worker health.
>> 
>> ~Gary
>> 
>> On 4/1/2015 7:52 PM, Harry Hoffman wrote:
>>> Hi folks,
>>> 
>>> So in my continuing pursuit of perfecting my Bro setup I found that adding a proxy on every box that also runs workers keeps bro much happier then a single manager/proxy box with one or more worker(s) boxes.
>>> 
>>> Prior to adding the additional proxies bro workers would die due to resource constraints.
>>> 
>>> Are other folks doing this?
>>> 
>>> Cheers,
>>> Harry
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> 
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> -- 
> Aashish Sharma	(asharma at lbl.gov) 				 
> Cyber Security, 
> Lawrence Berkeley National Laboratory  
> http://go.lbl.gov/pgp-aashish 
> Office: (510)-495-2680  Cell: (510)-612-7971
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro